←back to thread

AI models need a virtual machine

(blog.sigplan.org)
226 points azhenley | 3 comments | 30 Aug 25 13:25 UTC | HN request time: 0.001s | source
Show context
ayende ◴[30 Aug 25 15:59 UTC] No.45075710[source]▶
That is the wrong abstraction to think at. The problem is not _which_ tools you give the LLM, the problem is what action it can do.

For example, in the book-a-ticket scenario - I want it to be able to check a few websites to compare prices, and I want it to be able to pay for me.

I don't want it to decide to send me to a 37 hour trip with three stops because it is 3$ cheaper.

Alternatively, I want to be able to lookup my benefits status, but the LLM should physically not be able to provide me any details about the benefits status of my coworkers.

That is the _same_ tool cool, but in a different scope.

For that matter, if I'm in HR - I _should_ be able to look at the benefits status of employees that I am responsible for, of course, but that creates an audit log, etc.

In other words, it isn't the action that matters, but what is the intent.

LLM should be placed in the same box as the user it is acting on-behalf-of.

replies(9): >>45075916 #>>45076036 #>>45076097 #>>45076338 #>>45076688 #>>45077415 #>>45079715 #>>45080384 #>>45081371 #
nostrademons ◴[30 Aug 25 17:17 UTC] No.45076338[source]▶
What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are allowed to access, and there is physically no mechanism for them to do anything not on that list.

Unfortunately, no mainstream OS actually implements the capability model, despite some prominent research attempts [2], some half-hearted attempts at commercializing the concept that have largely failed in the marketplace [3], and some attempts to bolt capability-based security on top of other OSes that have also largely failed in the marketplace [4]. So the closest thing to capability-based security that is actually widely available in the computing world is a virtual machine, where you place only the tools that provide the specific capabilities you want to offer in the VM. This is quite imperfect - many of these tools are a lot more general than true capabilities should be - but again, modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace.

[1] https://en.wikipedia.org/wiki/Capability-based_security

[2] https://en.wikipedia.org/wiki/EROS_(microkernel)

[3] https://fuchsia.dev/

[4] https://sandstorm.io/

replies(4): >>45076969 #>>45077002 #>>45077449 #>>45080600 #
dbmikus ◴[30 Aug 25 19:44 UTC] No.45077449[source]▶
I'm going to be pedantic and note that iOS and Android both have the capability security model for their apps.

And totally agree that instead of reinventing the wheel here, we should just lift from how operating systems work, for two reasons:

1. there's a bunch of work and proven systems there already

2. it uses tools that exist in training data, instead of net new tools

replies(1): >>45077714 #
nostrademons ◴[30 Aug 25 20:23 UTC] No.45077714[source]▶
App permissions in iOS and Android are both too coarse-grained to really be considered capabilities. Capabilities (at least as they exist in something like Eros or Capsicum) are more "You have access to this specific file" or "You can make connections to this specific server" rather than "You have access to files" and "You have access to the network". The file descriptor is passed in externally from a privileged process where the user explicitly decides what rights to give the process; there is no open() or connect() syscall available to user programs.
replies(2): >>45078041 #>>45081910 #
1. saagarjha ◴[31 Aug 25 09:54 UTC] No.45081910[source]▶
This seems neat in theory but it is very difficult to actually do in practice. For example, let's say that you are allowed to make connections to a specific server. First, you have to get everyone onboard with isolating their code at that granularity, which requires a major rewrite that is easy to short-circuit by just being lazy and allowing overly broad permissions. But even then "a server" is hard to judge. Do you look at eTLD+1 (and ship PSL)? Do you look at IP (and find out that everything is actually Cloudflare)? Do you distinguish between an app that talks to the Gmail API, and one that is trying to reach Firebase for analytics? It's a hard problem. Most OSes do have some sort of capabilities as you've mentioned but the difficulty is not making them super fine-grained but actually designing them in a way that they have meaning.
replies(1): >>45086212 #
2. nostrademons ◴[31 Aug 25 19:22 UTC] No.45086212[source]▶
Yes, exactly. The implementation difficulties are why this idea hasn't taken the world by storm yet. Incentives are also not there for app developers to think about programming this way - it's much easier to just request a general permission and then work out the details later.

For the server ID, it really should be based on the public key of the server. A particular service should keep its private key secret, broadcast a public key for talking to it, and then being able to encrypt traffic that the server can decrypt defines a valid connection capability. Then the physical location of the server can change as needed, and you can intersperse layers of DDoS protection and load balancing and caching while being secure in knowing that all the intervening routers do not have access to the actual communication.

replies(1): >>45092125 #
3. saagarjha ◴[01 Sep 25 12:16 UTC] No.45092125[source]▶
Should Google and YouTube share a key? How about Google LLC and Google UK Limited?