The Rise of Whatever

Still don't get it. LLM outputs are nondeterministic. LLMs invent APIs that don't exist. That's why you filter those outputs through agent constructions, which actually compile code. The nondeterminism of LLMs don't make your compiler nondeterministic.

All sorts of ways to knock LLM-generated code. Most I disagree with, all colorable. But this article is based on a model of LLM code generation from 6 months ago which is simply no longer true, and you can't gaslight your way back to Q1 2024.

There is no value in randomly choosing an API that exists. There is value in choosing an API that works.

When LLM makes up an API that doesn't even exist it indicates that it's not tied to the reality of the task of fetching a working API, so filtering out nonexistent APIs will not help the other results match better. But yes, they'll compile.

LLMs can write truly shitty code. I have a <50% hit rate on stuff I don't have to rewrite, with Sketch.dev, an agent I like a lot. But they don't fail the way you or this article claim they do. Enough.