Something weird is happening with LLMs and chess

Keep in mind, everyone, that stockfish on its lowest level on lichess is absolutely terrible, and a 5-year old human who'd been playing chess for a few months could beat it regularly. It hangs pieces, does -3 blunders, totally random-looking bad moves.

But still, yes, something maybe a teeny tiny bit weird is going on, in the sense that only one of the LLMs could beat it. The arxiv paper that came out recently was much more "weird" and interesting than this, though. This will probably be met with a mundane explanation soon enough, I'd guess.

Here's a quick anonymous game against it by me, where I obliterate the poor thing in 11 moves. I was around a 1500 ELO classical strength player, which is, a teeny bit above average, globally. But I mean - not an expert, or even one of the "strong" club players (in any good club).

https://lichess.org/BRceyegK -- the game, you'll see it make the ultimate classic opening errors

https://lichess.org/ -- try yourself! It's really so bad, it's good fun. Click "play with computer" on the right, then level 1 is already selected, you hit go