GPT-4 and professional benchmarks: the wrong answer to the wrong question

I gave ChatGPT the four cards logic puzzle, lots of humans struggle with it, but chatGPT got it exactly right, so I was very impressed. Then I realised that the formulation I'd given it (the same as the original study) was almost certainly part of its training set.

I made extremely minor changes to the way the question was phrased and it failed badly, not just getting the answer wrong but falling into incoherence, claiming that T was a vowel, or that 3 was an even number.

The largeness of its training set can give an incorrect impression of its reasoning capabilities. It can apply simple logic to situations, even situations it hasn't seen, but the logic can't get much beyond the first couple of lectures in an introductory First-order Logic course before it starts to fall apart if it can't lean on its large training set data.

The fact that it can do logic at all is impressive to me though, I'm interested to see how much deeper its genuine capability goes as we get more advanced models.

GPT is a token prediction engine. It predicts what the next token is, and it does that very well. Its logical abilities are emergent and are limited by the design of the network. Transformers are constant-time computations: they compute a fixed number of steps and then they stop and produce a result. This is very different to how humans think, we can expend more time on a difficult task (sometimes years!), or give an instant answer to an easy task. And we have a conception of when a task is done, or when we have to think more.