The Rise of Whatever

AIs still frequently make up stuff up - there isn't really a way to get out of that. Have they improved a lot in the last six months? 100%! But they still make mistakes its quite common

LLM calls make stuff up. Your compiler can't make things up. An agent iterates LLM calls. When your LLM call makes an API up, your compiler will generate errors. The errors get fed back into the iterative loop. In pretty much ever real case, the LLM corrects, but either way: the result is clear. The code may be wrong, but it shouldn't hallucinate entire APIs.

But just compiling doesn't mean that much and doesn't really solve the core issue of AIs making stuff up. I could hook up a random word generator into a compiler and it also would also pass that test!

For example, just yesterday I asked an AI a question about how to approach a specific problem. It gave an answer that "worked" (it compiled!) but in reality it didn't really make any sense and would add a very nasty bug. What it wrote (It used a FrameUpdate instead of a normal Update) just didn't make sense on a basic level of how the framework worked.

I'm not interested in this Calvinball argument. The post we're commenting on makes a clear claim: an LLM hallucinating entire APIs. Not surreptitiously sneaking subtly shitty stuff past a compiler.

This is my problem: not that people are cynical about LLM-assisted coding, but that they themselves are hallucinating arguments about it, expecting their readers to nod along. Not happening here.

You can improve on that

1. A type-strict compiler.

2. https://github.com/isaacphi/mcp-language-server

LLMs will always make stuff up because they are lossy. In the same way that if I ask you to list the methods for some random object lib you'd not be able to do that; you use the documentation to pull that up or your code-complete companion. LLMs are just getting the tools for that.

A great solution to this problem, but it doesn't seem like this approach will generalize to problems in other fields, or even to more suble coding confabulations that can't be detected by the compiler or static analysis.

I vehemently agree with this. But it doesn't change the falsity of the claim in the article.

Oh for sure I agree 100%! I was just saying that they will always make stuff up no matter what. Those are both good fixes but at its core it can only "make stuff up".

> The post we're commenting on makes a clear claim: an LLM hallucinating entire APIs

You made a similar claim: LLMs invent APIs that don't exist

https://news.ycombinator.com/item?id=44461381

The AES block cipher core: also grievously insecure if used naively, without understanding what a block cipher can and can't do, by itself. Thus also an LLM call.