How to Build a Chess Engine and Fail

Why build a chess engine these days, just use an LLM?

Why consume when you can create?

Building is often more fun and almost always more educational than just using. Even in “failure”. :-)

Fun, a learning experience, practicing a skill set, curiosity, etc.

Take your pick.

LLMs are quite bad at chess actually, even compared to human players -- and certainly compared to proper modern chess engines

They usually struggle to always generate valid moves.

And that's pivotal.

If you have a program which always makes valid moves and gives up when it has lost you wrote a proper chess playing program. It may play badly, but it plays.

Your question sounds flippant but it’s actually quite deep. Why aren’t LLMs good at chess? The answer is likely that to be good at chess at some level requires search (and probably some ordering constraint on your evaluation function too that I’m not smart enough to figure out).

LLMs aren’t searching. They are memorizing. This explains their extremely poor performance on out of domain positions, whereas Stockfish can easily understand them.

That could now be achieved by precomputing all valid moves and using outlines[0] or Structured Outputs[1] to constrain the output.

[0]: https://github.com/dottxt-ai/outlines

[1]: https://openai.com/index/introducing-structured-outputs-in-t...

… or just a valid move checker that prompts it to try again if it fails to make a valid move.

I think chess is a great example to use to demonstrate that LLMs don’t actually know anything, and will be entirely unsuitable to replace humans on anything except thoroughly solved and documented problems.