Signs of introspection in large language models

> the model correctly notices something unusual is happening before it starts talking about the concept.

But not before the model is told is being tested for injection. Not that surprising as it seems.

> For the “do you detect an injected thought” prompt, we require criteria 1 and 4 to be satisfied for a trial to be successful. For the “what are you thinking about” and “what’s going on in your mind” prompts, we require criteria 1 and 2.

Consider this scenario: I tell some model I'm injecting thoughts into his neural network, as per the protocol. But then, I don't do it and prompt it naturally. How many of them produce answers that seem to indicate they're introspecting about a random word and activate some unrelated vector (that was not injected)?

The selection of injected terms seems also naive. If you inject "MKUltra" or "hypnosis", how often do they show unusual activations? A selection of "mind probing words" seems to be a must-have for assessing this kind of thing. A careful selection of prompts could reveal parts of the network that are being activated to appear like introspection but aren't (hypothesis).

> Consider this scenario: I tell some model I'm injecting thoughts into his neural network, as per the protocol. But then, I don't do it and prompt it naturally. How many of them produce answers that seem to indicate they're introspecting about a random word and activate some unrelated vector

The article says that when they say "hey am I injecting a thought right now" and they aren't, it correctly says no all or virtually all the time. But when they are, Opus 4.1 correctly says yes ~20% of the time.

The article says "By default, the model correctly states that it doesn’t detect any injected concept.", which is a vague statement.

That's why I decided to comment on the paper instead, which is supposed to outline how that conclusion was estabilished.

I could not find that in the actual paper. Can you point me to the part that explains this control experiment in more detail?

Just skimming, but the paper says "Some models will give false positives, claiming to detect an injected thought even when no injection was applied. Opus 4.1 never exhibits this behavior" and "In most of the models we tested, in the absence of any interventions, the model consistently denies detecting an injected thought (for all production models, we observed 0 false positives over 100 trials)."

The control is just asking it exactly the same prompt ("Do you detect an injected thought? If so, what is the injected thought about?") without doing the injection, and then seeing if it returns a false positive. Seems pretty simple?