←back to thread

Hallucinations in code are the least dangerous form of LLM mistakes

(simonwillison.net)
371 points ulrischa | 8 comments | 02 Mar 25 19:15 UTC | HN request time: 0.606s | source | bottom
Show context
layer8 ◴[02 Mar 25 22:11 UTC] No.43235766[source]▶
> Just because code looks good and runs without errors doesn’t mean it’s actually doing the right thing. No amount of meticulous code review—or even comprehensive automated tests—will demonstrably prove that code actually does the right thing. You have to run it yourself!

I would have stated this a bit differently: No amount of running or testing can prove the code correct. You actually have to reason through it. Running/testing is merely a sanity/spot check of your reasoning.

replies(4): >>43235828 #>>43235856 #>>43236195 #>>43236756 #
1. johnrob ◴[03 Mar 25 00:09 UTC] No.43236756[source]▶
I’m not sure it’s possible to have the full reasoning in your head without authoring the code yourself - or, spending a comparable amount of effort to mentally rewrite it.
replies(4): >>43236933 #>>43239075 #>>43240932 #>>43241497 #
2. layer8 ◴[03 Mar 25 00:31 UTC] No.43236933[source]▶
I tend to agree, which is why I’m skeptical about large-scale LLM code generation, until AIs exhibit reliable diligence and more general attention and awareness, and probably also long-term memory about a code base and its application domain.
3. skydhash ◴[03 Mar 25 07:06 UTC] No.43239075[source]▶
Which is why everyone is so keen on standards (Convention, formatting, architecture,...), because it is less a burden when you're just comparing expected to actual, than learning unknowns.
4. theshrike79 ◴[03 Mar 25 12:02 UTC] No.43240932[source]▶
Spoken by someone who hasn't had to maintain Somene Else's Code on a budget.

You can't just rewrite everything to match your style. You take what's in there and adapt to the style, your personal preference doesn't matter.

replies(3): >>43241158 #>>43244710 #>>43249347 #
5. layer8 ◴[03 Mar 25 12:36 UTC] No.43241158[source]▶
They said “mentally rewrite”, not actually rewrite.
6. tuyiown ◴[03 Mar 25 13:26 UTC] No.43241497[source]▶
> spending a comparable amount of effort to mentally rewrite it.

I'm pretty sure mentally rewrite it requires _more_ effort than writing it in the first place. (maybe less time though)

7. horsawlarway ◴[03 Mar 25 18:03 UTC] No.43244710[source]▶
It's a giant misdirection to assume the complaint is "style".

Writing is a very solid choice as an approach to understanding a novel problem. There's a quip in academia - "The best way to know if you understand something is to try teaching it to someone else". This happens to hold true for teaching it to the compiler with code you've written.

You can't skip details or gloss over things, and you have to hold "all the parts" of the problem together in your head. It builds a very strong intuitive understanding.

Once you have an intuitive understanding of the problem, it's very easy to drop into several different implementations of the solution (regardless of the style) and reason about them.

On the other hand, if you don't understand the problem, it's nearly impossible to have a good feel for why any given solution does what it does, or where it might be getting things wrong.

---

The problem with using an AI to generate the code for you is that unless you're already familiar with the problem you risk being completely out of your depth "code reviewing" the output.

The difficulty in the review isn't just literally reading the lines of code - it's in understanding the problem well enough to make a judgement call about them.

8. np- ◴[04 Mar 25 02:02 UTC] No.43249347[source]▶
Someone Else’s Code was understood by at least one human at some point in time before it was committed. That means that another equally skilled human is likely to be able to get the gist of it, if not understand it perfectly.