Hallucinations in code are the least dangerous form of LLM mistakes

Perhaps I'm just not that great of a coder, but I do have lots of code where if someone took a look it, it might look crazy but it really is the best solution I could find. I'm concerned LLMs won't do that, they won't take risks a human would or understand the implications of a block of code beyond its application in that specific context.

Other times, I feel like I'm pretty good at figuring out things and struggling in a time-efficient manner before arriving at a solution. LLM generated code is neat but I still have to spend similar amounts of time, except now I'm doing more QA and clean up work instead of debugging and figuring out new solutions, which isn't fun at all.

When people talk about 30% or 50% coding productivity gains with LLMs, I really want to know exactly what they're measuring.

[0] https://news.ycombinator.com/item?id=43236792

- keep the outline in my head: I don't give up the architect's seat. I decide which module does what and how it fits in the whole system, it's contract with other modules etc.

- review the code: this can be construed as negating the point of LLMs as this is time consuming but I think it is important to go through line by line and understand every line. You will absorb some of the LLM generated code in the process which will form an imperfect map in your head. That's essential for beginning troubleshooting next time things go wrong.

- last mile connectivity: several times the LLM takes you there but can't complete the last mile connectivity; instead of wasting time chasing it, do the final wiring yourself. This is a great shortcut to achieve the previous point.

puzzled. if you don't understand it fully, how can you say that it will look great to you, and that it will work?

Claude Code has been doing all of this for me on my latest project. It's remarkable.

It seems inevitable it'll get there for larger and more complex code bases, but who knows how far away that is.

Presumably, that simply reflects that a primary developer always has an advantage of having a more reliable understanding of a large code base - and the insights into the problem that come about during development challenges - than a reviewer of such code.

A lot of important bug subtle insights, many sub-verbal, into a problem come from going through the large and small challenges of creating something that solves it. Reviewers just don't get those insights as reliably.

Reviewers can't see all the subtle or non-obvious alternate paths or choices. They are less likely to independently identify subtle traps.

For AI code, that's a waste of time. The generated code will be based on an arbitrary patchwork of purposes and constraints, glued together well enough to function. I'm not saying it lacks purpose or constraints, it's just that those are inherited from random sources. The parts flow together with robotic but not human concern for consistency. It may incorporate brilliant solutions, but trying to infer intent or style or design philosophy is about as useful as doing handwriting analysis on a ransom note made from pasted-together newspaper clippings.

Both sorts of code have value. AI code may be well-commented. It may use features effectively that a human might have missed. Just don't try to anthropomorphize an AI coder or a lawnmower, you'll end up inventing an intent that doesn't exist.

This is what tests are for.

- generate - lint - format - fuzz - test - update

infintely?

Sorry, I'm failing to see your point.

Are you implying that the above is good enough, for a useful definition of good enough? I'm not disagreeing, and in fact that was my starting assumption in the message you're replying to.

Crap code can pass tests. Slow code can pass tests. Weird code can pass tests. Sometimes it's fine for code to be crap, slow, and/or weird. If that's your situation, then go ahead and use the code.

To expand on why someone might not want such code, think of your overall codebase as having a time budget, a complexity budget, a debuggability budget, an incoherence budget, and a maintenance budget. Yes, those overlap a bunch. A pile of AI-written code has a higher chance of exceeding some of those budgets than a human-written codebase would. Yes, there will be counterexamples. But humans will at least attempt to optimize for such things. AIs mostly won't. The AI-and-AI-using-human system will optimize for making it through your lint-fuzz-test cycle successfully and little else.

Different constraints, different outputs. Only you can decide whether the difference matters to you.

This is a degree of humility that made the scenario we are in much clearer.

Our information environment got polluted by the lack of such humility. Rhetoric that sounded ‘right’ is used everywhere. If it looks like an Oxford Don, sounds like an Oxford Don, then it must be an academic. Thus it is believable, even if they are saying the Titanic isn’t sinking.

Verification is the heart of everything humanity does, our governance structures, our judicial systems, economic systems, academia, news, media - everything.

It’s a massive computation effort to figure out what the best ways to allocate resources given current information, allowing humans to create surplus and survive.

This is why we dislike monopolies, or manipulations of these markets - they create bad goods, and screw up our ability to verify what is real.

How does doing the hard part provide a shortcut for reviewing all the LLVM code?

If anything it's a long cut, because now you have to understand the code and write it yourself. This isn't great, it's terrible.

Alas, I don't share your optimism about code I wrote myself. In fact, it's often harder to find flaws in my own code, then when reading someone else's code.

Especially if 'this is too complicated for me to review, please simplify' is allowed as a valid outcome of my review.

Just recently I think here on HN there was a discussion about how neural networks optimize towards the goal they are given, which in this case means exactly what you wrote, including that the code will do stuff in wrong ways just to pass the given tests.

Where do the tests come from? Initially from a specification of what "that thing" is supposed to do and also not supposed to do. Everyone who had to deal with specifications in a serious way knows how insanely difficult it is to get these right, because there are often things unsaid, there are corner cases not covered and so on. So the problem of correctness is just shifted, and the assumption that this may require less time than actually coding ... I wouldn't bet on it.

Conceptually the idea should work, though.

You don't know that though. There's no "it must work" criteria in the LLM training.

If you don’t understand it, ask the LLM to explain it. If you fail to get an explanation that clarifies things, write the code yourself. Don’t blindly accept code you don’t understand.

This is part of what the author was getting at when they said that it’s surfacing existing problems not introducing new ones. Have you been approving PRs from human developers without understanding them? You shouldn’t be doing that. If an LLM subsequently comes along and you accept its code without understanding it too, that’s not a new problem the LLM introduced.

At least when a human wrote it, someone understood the reasoning.

All of these can vary wildly in quality. Maybe its because I mostly use coding LLMs as either a research tool, or to write reasonably small and easy to follow chunks of code, but I find it no different than all of the other types of reading and understanding other people's code I already have to do.

‘Works for me’ isn’t actually _that_ useful a signal without serious qualification.

But the most troublesome to me is that it is just "pissing" out code and has no after-tough about the problem it is solving or the person it is talking to.

The number of times I have to repeat myself just to get a stubborn answer with no discussion is alarming. It does not benefit my well-being and is annoying to work with except for a bunch of exploratory cases.

I believe LLM are actually the biggest data heist organized. We believe that those models will get better at solving their jobs but the reality is that we are just giving away code, knowledge, ideas at scale, correcting the model for free, and paying to be allowed to do so. And when we watch the 37% minimum hallucination rate, we can more easily understand that the actual tough comes from the human using it.

I'm not comfortable having to argue with a machine and have to explain to it what I'm doing, how, and why - just to get it to spam me with things I have to correct afterwards anyway.

The worst is, all that data is the best insight on everything. How many people ask for X ? How much time did they spend trying to do X ? What were they trying to achieve ? Who are their customers ? etc...

"Short cuts make long delays." --Tolkien

I was appalled when I was being effusively thanked for catching some bugs in PRs. “No one really reads these,” is what I was told. Then why the hell do we have a required review?!

I think that just because someone might be more or less eloquent than someone else, the value of their thoughts and contributions shouldn't be weighed any differently. In a way, AI formatting and grammar assistance could be a step towards a more equitable future, one where ideas are judged on inherent merits rather than superficial junk like spel;ng or idk typos n shi.t

However, I think what the parent commenter (and I) might be saying is that it seems you're relying on AI for more than just help expressing yourself—it seems you're relying on it to do the thinking too. I'd urge you to consider if that's what you really want from a tool you use. That said, I'm just some random preachy-judgy stranger on the internet, you don't owe me shit, lol

(Side notes I couldn't help but include: I think talking about AI and language is way more complicated (and fascinating) than just that aspect, including things I'm absolutely unqualified to comment on—discrimination against AAVE use, classism, and racism can't and shouldn't be addressed by a magic-wand spell-checker that "fixes" everyone's speech to be "correct" (as if a sole cultural hegemony or way of speech is somehow better than any other))

> I think that just because someone might be more or less eloquent than someone else, the value of their thoughts and contributions shouldn't be weighed any differently. In a way, AI formatting and grammar assistance could be a step towards a more equitable future, one where ideas are judged on inherent merits rather than superficial junk like spel;ng or idk typos n shi.t

I guess I must come clean that my reply was sarcasm which obviously fell flat and caused you to come to the defense of those who can't spell - I swear I don't have anything against them.

> However, I think what the parent commenter (and I) might be saying is that it seems you're relying on AI for more than just help expressing yourself—it seems you're relying on it to do the thinking too. I'd urge you to consider if that's what you really want from a tool you use. That said, I'm just some random preachy-judgy stranger on the internet, you don't owe me shit, lol

You and presumably the parent commenter have missed the main point of the retort - you are assuming I am relying on AI for my content or its style. It is neither - I like writing point-wise in a systematic manner, always have, always will - AI or no-AI be damned. It is the all-knowing veil-piercing eagle-eyed deduction of random preachy-judgy strangers on the internet about something being AI-generated/aided just because it follows structure, that is annoying.

My solution for this is documentation, automated tests and sticking to the same conventions and libraries (like using Click for command line argument parsing) across as many projects as possible. It's essential that I can revisit a project and ramp up my mental model of how it works as quickly as possible.

I talked a bit more about this approach here: https://simonwillison.net/2022/Nov/26/productivity/

>(in other words, I can't prove the correctness ... without referencing the original paper).

agrees with what I said in my previous comment:

>if you don't understand it fully, how can you say .... that it will work?

(irrelevant parts from our original comments above, replaced with ... , without loss of meaning to my argument.)

both those quoted fragments, yours and mine, mean basically the same thing, i.e. that both you and the GP don't know whether it will work.

it's not that one cannot use some piece of code without knowing whether it works; everybody does that all the time, from algorithm books for example, as you said.

what you said just strengthens my argument.

testing can prove the presence of errors, but not their absence.

https://www.google.com/search?q=quote+testing+can+prove+the+...

- said by Steve McConnell (author of Code Complete), Edsger Dijkstra, etc. ...

it's what our lord and savior jesus christ uses for us humans if it is good for him its good enough for me. and surely google is not laying off 25k people because it believes humans are better than their LLMs :)

yes, and it sounds a bit like "works on my machine", a common cop-out which I am sure many of us have heard of.

google: works on my machine meme