Coding with LLMs in the summer of 2025 – an update

The software is largely there: you can run Ollama, vLLM or whatever else you please today.

The models are somewhat getting there: even the smaller ones like Qwen3-30B-A3B and Devstral-23B are okay for some use cases and can run decently fast. They’re not amazing, but better than much larger models a year or two ago.

The hardware is absolutely not there: most development laptops will be too weak to run a bunch of tools, IDEs and local services alongside a LLM and will struggle to do everything at the pace of those cloud services.

Even if you seek compromise and get a pair of Nvidia L4 cards or something similar and put them on a server somewhere, the aforementioned Qwen3-30B-A3B will run at around 60 tokens/second for a single query but slow down as you throw a bunch of developers at it that all need chat and autocomplete. The smaller Devstral model will more than halve the performance at the starting point because it’s dense.

Tools like GitHub Copilot allow an Ollama connection pretty easily, Continue.dev also does but can be a bit buggy (their VS Code implementation is better than their JetBrains one), whereas the likes of RooCode only seem viable with cloud models cause they generate large system prompts and need more performance than you can squeeze out of somewhat modest hardware.

That said, with more MoE models and better training, things seem hopeful. Just look at the recent ERNIE-4.5 release, their model is a bit smaller than Qwen3 but has largely comparable benchmark results.

Those Intel Arc Pro B60 cards can’t come soon enough. Someone needs to at least provide a passable alternative to Nvidia, nothing more.

And models like Qwen3 really don’t match the quality of Opus 4 and Gemini-Pro 2.5. And even if you manage to get your hands on 512GB of GPU RAM, it will be slow.

There’s simply so much going on under the hood at these LLM providers that are very hard to replicate locally.

Didn't Karpathy, in his latest talk, say something along the lines of: don't bother with less capable models, they are just a waste of time.

its impossible to catch up; but there is still much fertile and prospective territory within reach

It probably depends what your objective is. One of the benefits you get from running less capable models is that it's easier to understand what their limitations are. The shortcomings of more powerful models are harder to see and understand, because the models themselves are so much more capable.

If you have no interest in the inner workings of LLMs and you just want the machine to spit out some end result while putting in minimal time and effort, then yes, absolutely don't waste your time with smaller, less capable models.

On my M4 Max MacBook Pro, with MLX, I get around 70-100 tokens/sec for Qwen 3 30B-A3B (depending on context size), and around 40-50 tokens/sec for Qwen 3 14B. Of course they’re not as good as the latest big models (open or closed), but they’re still pretty decent for STEM tasks, and reasonably fast for me.

I have 128 GB RAM on my laptop, and regularly run multiple multiple VMs and several heavy applications and many browser tabs alongside LLMs like Qwen 3 30B-A3B.

Of course there’s room for hardware to get better, but the Apple M4 Max is a pretty good platform running local LLMs performantly on a laptop.

Is it really possible to learn from the mistakes of an LLM? It sounds like psychology or even alchemy 2.0, to be honest.

You can kinda get a feel for what they're good at, if you get what I mean?

Even the big online models have very specific styles and preferences for similar tasks. You can easily test this by giving them all some generic task without too many limits, each of them will gravitate towards a different solution to the same problem.