←back to thread

AI agents: Less capability, more reliability, please

(www.sergey.fyi)
419 points serjester | 2 comments | 31 Mar 25 14:45 UTC | HN request time: 0s | source
Show context
jedberg ◴[31 Mar 25 18:57 UTC] No.43538462[source]▶
I've been working on this problem for a while. There are whole companies that do this. They all work by having a human review a sample of the results and score them (with various uses of magic to make that more efficient). And then suggest changes to make it more accurate in the future.

The best companies can get up to 90% accuracy. Most are closer to 80%.

But it's important to remember, we're expecting perfection here. But think about this: Have you ever asked someone to book a flight for you? How did it go?

At least in my experience, there's usually a few back and forth emails, and then something is always not quite right or as good as if you did it yourself, but you're ok with that because it saved you time. The one thing that makes it better is if the same person does it for you a couple of times and learned your specific habits and what you care about.

I think the biggest problem in AI accuracy is expecting the AI to be better than a human.

replies(2): >>43538574 #>>43538649 #
1. lolinder ◴[31 Mar 25 19:14 UTC] No.43538649[source]▶
> I think the biggest problem in AI accuracy is expecting the AI to be better than a human.

If it's not better across at least one of {more accurate, faster, cheaper} then there is no business. You have to be offering one of the above.

And that applies both to humans and to existing tech solutions: an LLM solution must beat both in some dimension. Current flight booking interfaces are actually better than a human at all three: they're more accurate, they're free, and they're faster than trying to do the back and forth, which means the bar to clear for an agent is extremely high.

replies(1): >>43539391 #
2. bluGill ◴[31 Mar 25 20:14 UTC] No.43539391[source]▶
> Current flight booking interfaces are actually better than a human at all three

Only when you know exactly where to go. If you need to get to customers in 3 cities where order doesn't matter (ie the traveling salemen problem, though you are allowed to hit any city more than once) current solutions are not great. If you want to go on vacation but don't care much about where (almost every place with an airport would be an acceptable vacation, though some are better than others)