LLM Inevitabilism

Specific model I was using was o4-mini-high which the drop-down model selector describes as "Great at coding and visual reasoning".

Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though.

You can get the Google pro subscription (forget what they call it) that's ordinarily $20/mo for free right now (1 month free; can cancel whenever), which gives unlimited Gemini 2.5 Pro access.

I'm curious how you ended up in such a conversation in the first place. Hallucinations are one thing, but I can't remember the last time when the model was saying that it actually run something somewhere that wasn't a tool use call, or that it owns a laptop, or such - except when role-playing.

I wonder if the advice on prompting models to role play isn't backfiring now, especially in conversational setting. Might even be a difference between "you are an AI assistant that's an expert programmer" vs. "you are an expert programmer" in the prompt, the latter pushing it towards "role-playing a human" region of the latent space.

(But also yeah, o3. Search access is the key to cutting down on amount of guessing the answers, and o3 is using it judiciously. It's the only model I use for "chat" when the topic requires any kind of knowledge that's niche or current, because it's the only model I see can reliably figure out when and what to search for, and do it iteratively.)

What is really interesting is in the "thinking" section it said "I need to reassure the user..." so my intuition is that it thought it was right, but did not think I would think they were right, but if they just gave me the confidence, I would try the code and unblock myself. Maybe it thought this was the best % chance I would listen to it and so it is the correct response?

Yeah, this model didn't work it seems.

You're holding it wrong. You need to utter the right series of incantations to get some semblance of truth.

What, you used the model that was SOTA one week ago? Big mistake, that explains why.

You need to use this SOTA model that came out one day ago instead. That model definitely wasn't trained to overfit the week-old benchmarks and dismiss the naysayers. Look, a pelican!

What? You haven't verified your phone number and completed a video facial scan and passed a background check? You're NGMI.

Thank you for the tip on o3. I will switch to that and see how it goes. I do have a paid sub for ChatGPT, but from the dropdown model descriptions "Great at coding" sounded better than "Advanced reasoning". And 4 is like almost twice as much as 3.

I've seen that specific kind of role-playing glitch here and there with the o[X] models from openai. The models do kinda seem to just think of themselves as being developers with their own machines.. I think it usually just doesn't come up but can easily be tilted into it.

Maybe? Depends on what followed that thought process.

I've noticed this couple times with o3, too - early on, I'd catch a glimpse of something like "The user is asking X... I should reassure them that Y is correct" or such, which raised an eyebrow because I already know Y was bullshit and WTF with the whole reassuring business... but then the model would continue actually exploring the question and the final answer showed no trace of Y, or any kind of measurement. I really wish OpenAI gave us the whole thought process verbatim, as I'm kind of curious where those "thoughts" come from and what happens to them.

lol yep, fully get that. And I mean I'm sure o4 will be great but the '-mini' variant is weaker. Some of it will come down to taste and what kind of thing you're working on too but personal preferences aside, from the heavy LLM users I talk to o3 and gemini 2.5 pro at the moment seem to be top if you're dialoging with them directly (vs using through an agent system).

I’d also recommend basically always having search enabled. That’s eliminated major hallucinations for me.

In my current experience:

- o3 is the bestest and my go-to, but its strength comes from it combining reasoning with search - it's the one model you can count on finding things out for you instead of going off vibe and training data;

- GPT 4.5 feels the smartest, but also has tight usage limits and doesn't do search like o3 does; I use it when I need something creative done, or switch to it mid-conversation to have it reason off an already primed context;

- o4-mini / o4-mini-hard - data transformation, coding stuff that doesn't require looking things up - especially when o3 looked stuff up already, and now I just need ChatGPT to apply it into code/diagrams;

- gpt-4o - only for image generation, and begrudgingly when I run out of quota on GPT 4.5

o3 has been my default starting model for months now; most of my queries generally benefit from having a model that does autonomous reasoning+search. Agentic coding stuff, that I push to Claude Code now.

Ehh... I did ask it if it would be able to figure this out or if I should try another model :|

> Look, a pelican!

Love this reference :)

All LLMs can fail this way.

It's kind of weird to see people running into this kind of issue with modern large models with all the RL and getting confused. No one starting today seems to have good intuition for them. One person I knew insisted LLMs could do structural analysis for months until he saw some completely absurd output from one. This used to be super common with small GPTs from around 2022 and so everyone just intuitively knew to watch out for it.

Not saying this to defend the models as your point is fundamentally sound, but IIRC the user-visible "thoughts" are produced by another LLM summarising the real chain-of-thought, so weird inversions of what it's "really" "thinking" may well slip in at the user-facing level — the real CoT often uses completely illegible shorthand of its own, some of which is Chinese even when the prompt is in English, but even the parts in the users' own languages can be hard-to-impossible to interpret.

To agree with your point, even with the real CoT researchers have shown that model's CoT workspace don't accurately reflect behaviour: https://www.anthropic.com/research/reasoning-models-dont-say...

Okay. And the fact that LLMs routinely make up crap that doesn't exist but sounds plausible, and the fact that this appears to be a fundamental problem with LLMs, this doesn't give you any pause on your hype train? Genuine question, how do you reconcile this?

> I really wish OpenAI gave us the whole thought process verbatim, as I'm kind of curious where those "thoughts" come from and what happens to them.

Don't see what you mean by this; there's no such thing as "thoughts" of an LLM, and if you mean the feature marketers called chain-of-thought, it's yet another instance of LLMs making shit up, so.

> Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though.

I hope you appreciate just how crazy this sentence sounds, even in an age when this is normalised.

Yep, it's surreal.

I've heard my grandma talk about Catholic saints and their powers with a not dissimilar kind of discourse.

Point being?

Unlike Catholic saints, ChatGPT models actually exhibit these properties in directly observable and measurable way. I wrote how I decide which model to use for actual tasks, not which saint to pray to.

A friend recently had a similar interaction where ChatGPT told them that it had just sent them an email or a wetransfer with the requested file

the fact that one needs to know stuff like this and that it changes every three months seriously limits the usefulness of LLMs for me

My grandma also uses saints for actual tasks (e.g. St Anthony for finding lost items), and they exibith those properties in observable ways (e.g. he found her sewing needles just last month). Perhaps the comparison is more appropriate than you realise.

> actually exhibit these properties in directly observable and measurable way

Well but do they? I don't mean your vibes, and I also don't mean cooked-up benchmarks. For example: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

I get this. On the one hand, those things I wrote down are just simple conclusions from immediate experience, not something I had to learn or feel burdened by - but on the other hand, when I look at similar lists for e.g. how to effectively use Claude Code, I recoil in horror.

There's a silver lining in this, though: none of that is any kind of deep expertise, so there's no need for up-front investment. Just start using a tool and pay attention, and you'll pick up on those things in no time.

> Perhaps the comparison is more appropriate than you realise.

Or perhaps you stop being obtuse. There's no causal connection between "using saints for actual tasks" and the outcomes, which is why we call this religion. In contrast, you can see the cause-and-effect relationship directly and immediately with LLMs - all it takes is going to chatgpt.com or claude.ai, typing in a query, and observing the result.

> Well but do they? I don't mean your vibes, and I also don't mean cooked-up benchmarks.

Do read the study itself, specifically the parts where the authors spell out specifically what is or isn't being measured here.

> And the fact that LLMs routinely make up crap that doesn't exist but sounds plausible, and the fact that this appears to be a fundamental problem with LLMs, this doesn't give you any pause on your hype train? Genuine question, how do you reconcile this?

Simply. Because the same is the case with humans. Mostly for the same reasons.

(Are humans overhyped? Maybe?)

The LLM hype train isn't about them being more accurate or faster than what came before - it comes from them being able to understand what you mean. It's a whole new category of software - programs that can process natural language like humans would; a powerful side effect that took the world by surprise is, that making LLMs better at working with natural language implicitly turns them into general-purpose problem solvers.

> Don't see what you mean by this; there's no such thing as "thoughts" of an LLM, and if you mean the feature marketers called chain-of-thought, it's yet another instance of LLMs making shit up, so.

"Chain-of-thought" is so 2024; current models don't need to be told to "think step by step", they're post-trained to first generate a stream of intermediary tokens not meant as "actual answer", before continuing with the "actual answer". You can call it however you like; however both research literature and vendors settled on calling it "thinking" or "reasoning". Treat them as terms of art, if that helps.

It's really simple x) either the "observation" is just vibes, and then it's fundamentally the same as when Gran's knees get better after she asks Saint Euphemia, or it's actually a scientific observation, in which case please post! :)

You may not like but it's what it is.

Being in the cutting edge isn't for everyone. If you can find an island where staying updated is optional you can choose that. Imo, these islands are fast shrinking.

If not users opinions, or objective benchmarks, then what? Sounds like you prefer closing your ears saying 'nananana...'

I want to scream Jesus F Christ, but I guess that would be making your point for you :).

Still, it's not what you're saying. Vibes are present, too, but so are concrete effects like one-off utilities conjured in 2 minutes of prompting instead of 10 minutes of searching or 1 hour of coding, or research questions answered swiftly (and answers verified by me), that I normally wouldn't consider tackling due to up-front effort being greater than my discretionary time. And so on.

This is not vibes, but hard, empirical evidence, immediately retrievable as it just accumulates in the chat history, so I can always go back to check if I wasn't misremembering things.