Skyvern Browser Agent 2.0: How We Reached State of the Art in Evals

Many of the examples given for agents such as this are things I just flat wouldn’t trust an LLM to do - buying something on Amazon for example: Will it pick new or ‘renewed’? Will it select an item that is from a janky looking vendor and may be counterfeit? Will it pick the cheapest option for me? What if multiple colors are offered?

This one example alone has so many branches that would require knowing what’s in my head.

On the flip side, it’s a ridiculously simple task for a human to do for themselves, so what am I truly saving?

Call me when I can ask it to check the professional reviews of X category on N websites (plus YouTube), summarize them for me, and find the cheapest source for the top 2 options in the category that will arrive in Y days or sooner.

That would be useful.

I got Amazon Prime. If it has Prime, it is a no-brainer. Free return for 30 days. No S&H costs. Only cost is my time.

Yea, but LLMs cannot reason - we've all seen them blurt out complete non-sequitur, or end up in death loops of pseudo-reasoning (e.g. https://news.ycombinator.com/item?id=42734681 has a few examples). I don't think one should trust an LLM to pick Prime products all the time even if that's very explicitly requested - I'm sure it's possible to minimize errors so it'll do the right thing most of the time, but having a guarantee that it won't pick non-Prime item sounds impossible. Same for any other tasks - if there is a way to make a mistake, a mistake will be eventually made.

(Idk if we can trust a human either - brain farts are a thing after all, but at least humans are accountable. Machines are not - at least not at the moment.)

You cannot trust a human to avoid buying crap on Amazon but like I said with Prime the only cost is time (and, to be fair: Co2 footprint).

Dynamic CVV would mean you'd have to authorize the payment. If amount seems off, decline.

To be clear, I don't think I'd use it but if it could save you time (a precious value, in our day and age) with good signal to noise ratio it is win-win for user, author, and Amazon.

If you want to buy an Apple device from a trusted party, including trusted accessories, then there's apple.com. My point being: buying from there is much more secure. But even then, there is no '1 iPhone 16'; there's variants. Many of them.

If it goes on your skin or in your body, do you trust Amazon 3rd party sellers?