Some thoughts on LLMs and software development

In that framing, you can look at an agent as simply a filter on those hallucinations.

Yes yes, with yet to be discovered holes

Isn't an "agent" not just hallucinations layered on top of other random hallucinations to create new hallucinations?

No, that's exactly what an agent isn't. What makes an agent an agent is all the not-LLM code. When an agent generates Golang code, it runs the Go compiler, which is in the agent's architecture an extension of the agent. The Go compiler does not hallucinate.

This vaguely relates to a theory about human thought: that our subconscious constantly comes up with random ideas, then filters the unreasonable ones, but in people with delusions (e.g. schizophrenia) the filter is broken.

Salience (https://en.wikipedia.org/wiki/Salience_(neuroscience)), "the property by which some thing stands out", is something LLMs have trouble with. Probably because they're trained on human text, which ranges from accurate descriptions of reality to nonsense.

The most common "agent" is an letting an LLM run a while loop (“multi-step agent”) [1]

[1] https://huggingface.co/docs/smolagents/conceptual_guides/int...

That's not how Claude Code works (or Gemini, Cursor, or Codex).

More of a error-correcting feedback loop rather than a filter, really. Which is very much what we do as humans, apparently. One recent theory of neuroscience that is becoming influential is Predictive Processing --https://en.wikipedia.org/wiki/Predictive_coding -- this postulates that we also constantly generate a "mental model" of our environment (a literal "prediction") and use sensory inputs to correct and update it.

So the only real difference between "perception" and a "hallucination" is whether it is supported by physical reality.

thats a fascinating way to put it

You ever see something out of the corner of your eye and you were mistaken? It feels like LLM hallucinations. "An orange cat on my counter?!" And an instant later, your brain has reclassified "a basketball on my counter" as that fits the environment model better as several instant-observations gather contexts: not moving, more round, not furry, I don't own a cat, yesterday my kid mentioned something about tryouts, boop insta-reclassification from cat to basketball.

I can recognize my own meta cognition there. My model of reality course corrects the information feed interpretation on the fly. Optical illusions feel very similar whereby the inner reality model clashes with the observed.

For general ai, it needs a world model that can be tested against and surprise is noted and models are updated. Looping llm output with test cases is a crude approximation of that world model.

Sounds like a kalman filter, which suggests to me that it’s too simplistic a perspective.

I leaned heavily on my own meta cognition recently when withdrawing from a bereavment benzo habit recently. The combo of flu symptoms, anxiety and hallucinations are fierce. I knew O was seeing things that were not real; light fittings turning into a rotating stained glass slideshow. So I'm totally on board with the visual model hypothesis. My own speculation is that audio perception is less predictive as audio structure persists deeper into my own DMT sessions than does vision, where perspective quickly collapses and vision becomes kaleidoscopic. Which may be a return to the vision we had as newborns. Maybe normal vision is only attained via socialisation?