I Self-Hosted Llama 3.2 with Coolify on My Home Server

Probably saves a bit on the gas bill for heating too

If only we had heat-pump computers

I don’t know, it’s kind of amazing how good the lighter weight self hosted models are now.

Given a 16gb system with cpu inference only, I’m hosting gemma2 9b at q8 for llm tasks and SDXL turbo for image work and besides the memory usage creeping up for a second or so while i invoke a prompt, they’re basically undetectable in the background.

Snark aside, even in Germany (where electricity is very expensive) it is more economical to self host than to pay for a subscription to any of the commercial providers.

I'd gladly run whatever model you want at home, rent it out so you can pay for both heating, the GPU and the power consumed :-)