←back to thread

The security paradox of local LLMs

(quesma.com)
146 points jakozaur | 2 comments | 22 Oct 25 12:48 UTC | HN request time: 0.404s | source
Show context
AnthonyMouse ◴[22 Oct 25 19:48 UTC] No.45674236[source]▶
Everybody is talking about how this is obvious or not a real problem, but I think the flaw in it is something else.

It assumes that local models are inherently worse. But from a software perspective that's nonsense because there is no reason it couldn't be the exact same software. And from a hardware perspective the theory would have to be that the centralized system is using more expensive hardware, but there are two ways around that. The first is that you can sacrifice speed for cost -- x86 servers are slower than GPUs but can run huge models because they support TBs of memory. And the second is that you can, of course, buy high end local hardware, as many enterprises might choose to do, especially when they have enough internal users to keep it busy.

replies(1): >>45674427 #
1. turtletontine ◴[22 Oct 25 20:04 UTC] No.45674427[source]▶
The point (which they make quite explicitly) is that an individual or small organization can only run open source models locally, and those open source models are less sophisticated than the “frontier” models.

Obviously we can’t run GPT-5 or the cutting edge version of Claude or whatever locally, because OpenAI or Anthropic are keeping those weights as closely kept secrets.

replies(1): >>45674857 #
2. AnthonyMouse ◴[22 Oct 25 20:39 UTC] No.45674857[source]▶
But there is nothing inherent about that. The companies that want to run local models, or the cloud and hardware providers that want to sell hardware to run them, can get together and publish better local models.

Moreover, even that's presuming that you would only use the best available model, but that's also likely to be the one which is the most resource intensive and the most expensive, and then you can't afford it anyway. Meanwhile to use their smaller models you're still paying their margin, whereas if you use a local model you can spend that money on hardware. The bigger local model can beat the smaller proprietary one for the same price.