←back to thread

I Self-Hosted Llama 3.2 with Coolify on My Home Server

(geek.sg)
221 points whitefables | 1 comments | 16 Oct 24 05:26 UTC | HN request time: 0.204s | source
Show context
ragebol ◴[16 Oct 24 08:13 UTC] No.41856737[source]▶
Probably saves a bit on the gas bill for heating too
replies(3): >>41856985 #>>41856987 #>>41856994 #
1. CraigJPerry ◴[16 Oct 24 08:53 UTC] No.41856987[source]▶
I don’t know, it’s kind of amazing how good the lighter weight self hosted models are now.

Given a 16gb system with cpu inference only, I’m hosting gemma2 9b at q8 for llm tasks and SDXL turbo for image work and besides the memory usage creeping up for a second or so while i invoke a prompt, they’re basically undetectable in the background.