Some thoughts on LLMs and software development

> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.

Nice.

In that framing, you can look at an agent as simply a filter on those hallucinations.

Yes yes, with yet to be discovered holes

Nah I don't agree with this characterization. The problem is, the majority of those hallucinations are true. What was said would make more sense if the majority of the responses were, in fact, false, but this is not the case.

Isn't an "agent" not just hallucinations layered on top of other random hallucinations to create new hallucinations?

I think you're both correct but have different definitions of hallucinations. You're judging it as a hallucination based on the veracity of the output. Whereas Fowler is judging it based on the method by which the output is achieved. By that judgement, everything is a hallucination because the user cannot differentiate between when the LLM is telling the truth and isn't.

This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.

No, that's exactly what an agent isn't. What makes an agent an agent is all the not-LLM code. When an agent generates Golang code, it runs the Go compiler, which is in the agent's architecture an extension of the agent. The Go compiler does not hallucinate.

This vaguely relates to a theory about human thought: that our subconscious constantly comes up with random ideas, then filters the unreasonable ones, but in people with delusions (e.g. schizophrenia) the filter is broken.

Salience (https://en.wikipedia.org/wiki/Salience_(neuroscience)), "the property by which some thing stands out", is something LLMs have trouble with. Probably because they're trained on human text, which ranges from accurate descriptions of reality to nonsense.

an LLM hallucination is defined by its truth

> In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called confabulation,[1] or delusion)[2] is a response generated by AI that contains false or misleading information presented as fact.[3][4]

You say

> This is different from human hallucinations where it makes something up because of something wrong with the mind rather than some underlying issue with the brain's architecture.

For consistency you might as well say everything the human mind does is hallucination. It's the same sort of claim. This claim at least has the virtue of being taken seriously by people like Descartes.

https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...

Even the colloquial term outside of AI is characterized by the veracity of the output.

The most common "agent" is an letting an LLM run a while loop (“multi-step agent”) [1]

[1] https://huggingface.co/docs/smolagents/conceptual_guides/int...

I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and three is usually said to be larger than two, so they can speak in a way that mostly does not contradict this knowledge. They can use tools to count, like a human who knows digits would use a calculator.

But much more than an arithmetic engine, the current crop of AI needs an epistemic engine, something that would help follow logic and avoid contradictions, to determine what is a well-established fact, and what is a shaky conjecture. Then we might start trusting the AI.

That's not how Claude Code works (or Gemini, Cursor, or Codex).

Isn’t that why people argue against calling them hallucinations?

It implies that some parts of the output aren’t hallucinations, when the reality is that none of it has any thought behind it.

this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works, not even the builders, here we are.

LLMs don't hallucinate : they bullshit (which is not caring about truth).

I have a very similar (probably unoriginal) thought about some human mental illnesses.

So, we VALUE creativity, we claim that it helps us solve problems, improves our understanding of the universe, etc.

BUT people with some mental illnesses, their brain is so creative that they lose the understanding of where reality is and where their imagination/creativity takes over.

eg. Hearing voices? That's the brain conjuring up a voice - auditory and visual hallucinations are the easy example.

But it goes further, depression is where people's brains create scenarios where there is no hope, and there's no escape. Anxiety too, the brain is conjuring up fears of what's to come

This isn't a good characterization of it either. I don't think LLMs know the difference. Bullshit implies they are lying.

It's possible LLMs are lying but my guess is that they really just can't tell the difference.

I've prefered to riff off of the other quote:

"All (large language) model outputs are hallucinations, but some are useful."

Some astonishingly large proportion of them, actually. Hence the AI boom.

More of a error-correcting feedback loop rather than a filter, really. Which is very much what we do as humans, apparently. One recent theory of neuroscience that is becoming influential is Predictive Processing --https://en.wikipedia.org/wiki/Predictive_coding -- this postulates that we also constantly generate a "mental model" of our environment (a literal "prediction") and use sensory inputs to correct and update it.

So the only real difference between "perception" and a "hallucination" is whether it is supported by physical reality.

"Since no one actually knows how all of it works, not even the builders, here we are."

To me this is the most bizarre part. Have we ever had a technology deployed at this scale without a true understanding of its inner workings?

My fear is that the general public perception of AI will be damaged since for most LLMs = AI.

Are you sure you're talking about LLMs? These sound more like traditional ML systems like AlphaFold or AlphaProof.

Humanity used fire for like a bazillion years before figuring out thermodynamics

This is a misconception, we absolutely do know how LLMs work, that's how we can write them and publish research papers.

The idea we don't is tabloid journalism, it's simply because the output is (usually) randomised - taken to mean, by those who lack the technical chops, that programmers "don't know how it works" because the output is indeterministic.

This is not withstanding we absolutely can repeat the output by using not randomisation (temperature 0).

thats a fascinating way to put it

They're using the term 'bullshit' as it is understood as a term of art, which doesn't imply lying. It's closer to creating a response without any regard for telling the truth. Bullshitting is often most effective when you happen to be telling the truth, although the bullshitter has no commitment to that.

You ever see something out of the corner of your eye and you were mistaken? It feels like LLM hallucinations. "An orange cat on my counter?!" And an instant later, your brain has reclassified "a basketball on my counter" as that fits the environment model better as several instant-observations gather contexts: not moving, more round, not furry, I don't own a cat, yesterday my kid mentioned something about tryouts, boop insta-reclassification from cat to basketball.

I can recognize my own meta cognition there. My model of reality course corrects the information feed interpretation on the fly. Optical illusions feel very similar whereby the inner reality model clashes with the observed.

For general ai, it needs a world model that can be tested against and surprise is noted and models are updated. Looping llm output with test cases is a crude approximation of that world model.

Sounds like a kalman filter, which suggests to me that it’s too simplistic a perspective.

I leaned heavily on my own meta cognition recently when withdrawing from a bereavment benzo habit recently. The combo of flu symptoms, anxiety and hallucinations are fierce. I knew O was seeing things that were not real; light fittings turning into a rotating stained glass slideshow. So I'm totally on board with the visual model hypothesis. My own speculation is that audio perception is less predictive as audio structure persists deeper into my own DMT sessions than does vision, where perspective quickly collapses and vision becomes kaleidoscopic. Which may be a return to the vision we had as newborns. Maybe normal vision is only attained via socialisation?

You may like to check out Iain McGilchrist's take on schizophrenia, which essentially he says is a relative excess of rationality ("if then else" thinking) and a deficit of reasonableness (as in sensible context inhabiting).

One night, I asked it to write me some stories, it did seem happy doing that. I just kept saying do what you want when it asked me for a choice, its a fun little way to spend a couple of hours.

I've asked LLMs to modify 5 files at a time and mark a checklist. Also image generators no longer draw 6 fingered hands.

I find it a bit of a reductive way of looking at it personally

I disagree. IF something is bullshit nobody means that bullshit is possibly true. Bullshit is colloquially always false and always a lie.

I shall have a read of that at some point.