←back to thread

Ask HN: What's Your Useful Local LLM Stack?

91 points Olshansky | 1 comments | 15 Jul 25 15:17 UTC | HN request time: 0s | source

What I’m asking HN:

What does your actually useful local LLM stack look like?

I’m looking for something that provides you with real value — not just a sexy demo.

---

After a recent internet outage, I realized I need a local LLM setup as a backup — not just for experimentation and fun.

My daily (remote) LLM stack:

  - Claude Max ($100/mo): My go-to for pair programming. Heavy user of both the Claude web and desktop clients.

  - Windsurf Pro ($15/mo): Love the multi-line autocomplete and how it uses clipboard/context awareness.

  - ChatGPT Plus ($20/mo): My rubber duck, editor, and ideation partner. I use it for everything except code.
Here’s what I’ve cobbled together for my local stack so far:

Tools

  - Ollama: for running models locally

  - Aider: Claude-code-style CLI interface

  - VSCode w/ continue.dev extension: local chat & autocomplete
Models

  - Chat: llama3.1:latest

  - Autocomplete: Qwen2.5 Coder 1.5B

  - Coding/Editing: deepseek-coder-v2:16b
Things I’m not worried about:

  - CPU/Memory (running on an M1 MacBook)

  - Cost (within reason)

  - Data privacy / being trained on (not trying to start a philosophical debate here)
I am worried about:

  - Actual usefulness (i.e. “vibes”)

  - Ease of use (tools that fit with my muscle memory)

  - Correctness (not benchmarks)

  - Latency & speed
Right now: I’ve got it working. I could make a slick demo. But it’s not actually useful yet.

---

Who I am

  - CTO of a small startup (5 amazing engineers)

  - 20 years of coding (since I was 13)

  - Ex-big tech
Show context
ashwinsundar ◴[15 Jul 25 16:48 UTC] No.44573186[source]▶
I just go outside when my internet is down for 15 minutes a year. Or tether to my cell phone plan if the need is urgent.

I don't see the point of a local AI stack, outside of privacy or some ethical concerns (which a local stack doesn't solve anyway imo). I also *only* have 24GB of RAM on my laptop, which it sounds like isn't enough to run any of the best models. Am I missing something by not upgrading and running a high-performance LLM on my machine?

replies(1): >>44573265 #
filchermcurr ◴[15 Jul 25 16:55 UTC] No.44573265[source]▶
I would say cost is a factor. Maybe not for OP, but many people aren't able to spend $135 a month on AI services.
replies(1): >>44573407 #
ashwinsundar ◴[15 Jul 25 17:05 UTC] No.44573407[source]▶
Does the cost of a new computer not get factored in? I think I would need to spend $2000+ to run a decent model locally, and even then I can only run open source models

Not to mention, running a giant model locally for hours a day is sure to shorten the lifespan of the machine…

replies(3): >>44573609 #>>44573634 #>>44574438 #
dpoloncsak ◴[15 Jul 25 17:22 UTC] No.44573609[source]▶
$2000 for a new machine is only a little over a year in AI costs for OP
replies(1): >>44580499 #
lm28469 ◴[16 Jul 25 09:59 UTC] No.44580499{3}[source]▶
Electricity isn't free and these things are basically continuously ON bread toasters.
replies(2): >>44582861 #>>44595430 #
1. trod1234 ◴[17 Jul 25 16:55 UTC] No.44595430{4}[source]▶
It is effectively free when you have a surplus of E- coming from the sun. (PV)