Hallucinations in code are the least dangerous form of LLM mistakes

(simonwillison.net)

Show context

notepad0x90 ◴[02 Mar 25 23:24 UTC] No.43236385[source]▶

My fear is that LLM generated code will look great to me, I won't understand it fully but it will work. But since I didn't author it, I wouldn't be great at finding bugs in it or logical flaws. Especially if you consider coding as piecing together things instead of implementing a well designed plan. Lots of pieces making up the whole picture but a lot of those pieces are now put there by an algorithm making educated guesses.

Perhaps I'm just not that great of a coder, but I do have lots of code where if someone took a look it, it might look crazy but it really is the best solution I could find. I'm concerned LLMs won't do that, they won't take risks a human would or understand the implications of a block of code beyond its application in that specific context.

Other times, I feel like I'm pretty good at figuring out things and struggling in a time-efficient manner before arriving at a solution. LLM generated code is neat but I still have to spend similar amounts of time, except now I'm doing more QA and clean up work instead of debugging and figuring out new solutions, which isn't fun at all.

replies(13): >>43236847 #>>43237043 #>>43237101 #>>43237162 #>>43237387 #>>43237808 #>>43237956 #>>43238722 #>>43238763 #>>43238978 #>>43239372 #>>43239665 #>>43241112 #

1. fuzztester ◴[03 Mar 25 01:00 UTC] No.43237162[source]▶

>>43236385 #

>My fear is that LLM generated code will look great to me, I won't understand it fully but it will work.

puzzled. if you don't understand it fully, how can you say that it will look great to you, and that it will work?

replies(4): >>43237362 #>>43238789 #>>43241331 #>>43247810 #

2. Nevermark ◴[03 Mar 25 01:32 UTC] No.43237362[source]▶

>>43237162 (TP) #

> if you don't understand it fully, how can you say that it will look great to you, and that it will work?

Presumably, that simply reflects that a primary developer always has an advantage of having a more reliable understanding of a large code base - and the insights into the problem that come about during development challenges - than a reviewer of such code.

A lot of important bug subtle insights, many sub-verbal, into a problem come from going through the large and small challenges of creating something that solves it. Reviewers just don't get those insights as reliably.

Reviewers can't see all the subtle or non-obvious alternate paths or choices. They are less likely to independently identify subtle traps.

3. raincole ◴[03 Mar 25 06:04 UTC] No.43238789[source]▶

>>43237162 (TP) #

It happens all the time. Way before LLM. There were countless times I implemented an algorithm from a paper or a book while not fully understanding it (in other words, I can't prove the correctness or time complexity without referencing the original paper).

replies(1): >>43245703 #

4. rsynnott ◴[03 Mar 25 13:03 UTC] No.43241331[source]▶

>>43237162 (TP) #

I mean, depends what you mean by ‘work’. For instance, something which produces the correct output, and leaks memory, is that working? Something which produces the correct output, but takes a thousand times longer than it should; is that working? Something which produces output which looks superficially correct and passes basic tests, is that working?

‘Works for me’ isn’t actually _that_ useful a signal without serious qualification.

replies(2): >>43247005 #>>43275331 #

5. fuzztester ◴[03 Mar 25 19:30 UTC] No.43245703[source]▶

>>43238789 #

imo, your last phrase, excerpted below:

>(in other words, I can't prove the correctness ... without referencing the original paper).

agrees with what I said in my previous comment:

>if you don't understand it fully, how can you say .... that it will work?

(irrelevant parts from our original comments above, replaced with ... , without loss of meaning to my argument.)

both those quoted fragments, yours and mine, mean basically the same thing, i.e. that both you and the GP don't know whether it will work.

it's not that one cannot use some piece of code without knowing whether it works; everybody does that all the time, from algorithm books for example, as you said.

6. fuzztester ◴[03 Mar 25 21:29 UTC] No.43247005[source]▶

>>43241331 #

exactly.

what you said just strengthens my argument.

7. upcoming-sesame ◴[03 Mar 25 22:52 UTC] No.43247810[source]▶

>>43237162 (TP) #

You write a decent amount of tests

replies(1): >>43248938 #

8. fuzztester ◴[04 Mar 25 01:08 UTC] No.43248938[source]▶

>>43247810 #

Famous quote, read many years ago:

testing can prove the presence of errors, but not their absence.