AI models need a virtual machine

That is the wrong abstraction to think at. The problem is not _which_ tools you give the LLM, the problem is what action it can do.

For example, in the book-a-ticket scenario - I want it to be able to check a few websites to compare prices, and I want it to be able to pay for me.

I don't want it to decide to send me to a 37 hour trip with three stops because it is 3$ cheaper.

Alternatively, I want to be able to lookup my benefits status, but the LLM should physically not be able to provide me any details about the benefits status of my coworkers.

That is the _same_ tool cool, but in a different scope.

For that matter, if I'm in HR - I _should_ be able to look at the benefits status of employees that I am responsible for, of course, but that creates an audit log, etc.

In other words, it isn't the action that matters, but what is the intent.

LLM should be placed in the same box as the user it is acting on-behalf-of.

What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are allowed to access, and there is physically no mechanism for them to do anything not on that list.

Unfortunately, no mainstream OS actually implements the capability model, despite some prominent research attempts [2], some half-hearted attempts at commercializing the concept that have largely failed in the marketplace [3], and some attempts to bolt capability-based security on top of other OSes that have also largely failed in the marketplace [4]. So the closest thing to capability-based security that is actually widely available in the computing world is a virtual machine, where you place only the tools that provide the specific capabilities you want to offer in the VM. This is quite imperfect - many of these tools are a lot more general than true capabilities should be - but again, modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace.

[1] https://en.wikipedia.org/wiki/Capability-based_security

[2] https://en.wikipedia.org/wiki/EROS_(microkernel)

[3] https://fuchsia.dev/

[4] https://sandstorm.io/

I'm going to be pedantic and note that iOS and Android both have the capability security model for their apps.

And totally agree that instead of reinventing the wheel here, we should just lift from how operating systems work, for two reasons:

1. there's a bunch of work and proven systems there already

2. it uses tools that exist in training data, instead of net new tools

App permissions in iOS and Android are both too coarse-grained to really be considered capabilities. Capabilities (at least as they exist in something like Eros or Capsicum) are more "You have access to this specific file" or "You can make connections to this specific server" rather than "You have access to files" and "You have access to the network". The file descriptor is passed in externally from a privileged process where the user explicitly decides what rights to give the process; there is no open() or connect() syscall available to user programs.