AI agents: Less capability, more reliability, please

Yeah, the "book a flight" agent thing is a running joke now - it was a punchline in the Swyx keynote for the recent AI Engineer event in NYC: https://www.latent.space/p/agent

I think this piece is underestimating the difficulty involved here though. If only it was as easy as "just pick a single task and make the agent really good at that"!

The problem is that if your UI involves human beings typing or talking to you in a human language, there is an unbounded set of ways things could go wrong. You can't test against every possible variant of what they might say. Humans are bad at clearly expressing things, but even worse is the challenge of ensuring they have a concrete, accurate mental model of what the software can and cannot do.

Case-in-point look how long it’s taken for self-driving cars to mature. And many would argue they still have a ways to go until they’re truly reliable.

I think this highlights how we still haven’t cracked intelligence. Many of these issues come from the model’s very limited ability to adapt on the fly.

If you think about it every little action we take is a micro learning opportunity. A small-scale scientific process of trying something and seeing the result. Current AI models can’t really do that.

Even maps. I was driving to Chicago last week and Apple Maps insisted I take the exit for Danville. Fortunately I knew better, I only had the map on in case an accident might require rerouting. I find it hard to drive with maps navigation because they are usually correct, but wrong often enough that I don't fully trust them. So I have to double check everything they tell me with the reality in front of me, and that takes more mental effort than it ideally should.