I feel like the article neglects one obvious possibility: that OpenAI decided that chess was a benchmark worth "winning", special-cases chess within gpt-3.5-turbo-instruct, and then neglected to add that special-case to follow-up models since it wasn't generating sustained press coverage.
I suspect the same thing. Rather than LLMs “learning to play chess,” they “learnt” to recognise a chess game and hand over instructions to a chess engine. If that’s the case, I don’t feel impressed at all.
Recognize and hand over to a specialist engine? That might be useful for AI. Maybe I am missing something.
It is and would be useful, but it would be quite a big lie to the public, but more importantly to paying customers, and even more importantly to investors.