Grandmaster-level chess without search

OT: what's the state of the art in non-GM level computer chess?

Say I want to play chess with an opponent that is at about the same skill level as me, or perhaps I want to play with an opponent about 100 rating points above me for training.

Most engines let you dumb them down by cutting search depth, but that usually doesn't work well. Sure, you end up beating them about half the time if you cut the search down enough but it generally feels like they were still outplaying you for much of the game and you won because they made one or two blunders.

What I want is a computer opponent that plays at a level of my choosing but plays a game that feels like that of a typical human player of that level.

Are there such engines?

No, not with adjustable rating. The best human-like engine is fairymax, but its Elo is estimated between 1700-2000.

It doesn't seem that difficult to pull off - take one of the existing engines, get the top y moves, choose randomly. For each level down increase y by 1.

It doesn't work that way. There are many positions with lots of moves that are reasonable, but many others with only 1-2 sensible moves. It would make lots of obvious blunders that an amateur human would never make.

Maia does this reasonably well! You can play against it on Lichess. I have gotten a few "feels like a human" moments when playing against it - for example, getting it to fall into a trap that could trick a human but would easily be seen by a traditional search algorithm. It's not adjustable but there are a few different versions with different ratings (although it's not a very wide range).

https://www.maiachess.com/

https://lichess.org/@/maia1

Seems this might still have the problem of moves being either extremely good or extremely bad depending on how many good moves are found, rather than playing at a consistent level. Or for example in a degenerate case where there are only two moves and one leads to mate, the computer will be picking randomly.

Your engine would only mate once it had y options to mate.

Also attention. Lower level human players are more likely to make a move close to their own/their opponent's recent move. They're focused on one area of the board.

Basic computer opponents on the other hand can make moves all over the place. They look at the board state holistically. This can be very frustrating to play against as a human who has enough problems just thinking their way through some subset of the board, but is thrown off by the computer again and again.

It's not that bad in chess at least (compared to Go), but still something worth to keep in mind if you're trying to make an AI that is fun to play against as an amateur.

No, it doesn't work at all. Human mistakes are not at all like computer mistakes. Like -- blundering a piece in a 1-2 move combination will straight up never show up in the stockfish move tree, no matter what you set `y` to.

Or y chances. They did say it’d pick randomly. Still not great though, if a bit less funny.

GPT-3.5-turbo-instruct has a peak Elo of around 1800 (but of course can be prompted to play with less skill) and is as human like as you'll get at that level.

Piggy-backing off this - does anyone know of a quick way to evaluate the maia weights from python or js for a single board state? I'm trying to hack something together with my own search func intended for human play and I can't quite figure it out from the cpp in Lc0.

I built something like this. It works as long as you're not too high-rated: chessmate.ai. Once players get higher rated it is more difficult to predict their moves because you need to model their search process, not just their intuitive move choice. It's also possible to train on one player's games only so that it is more personalized.

It uses a similar approach to Maia but with a different neural network, so it had a bit better move matching performance. And on top of that it has an expectation maximization algorithm so that the bot will try to exploit your mistakes.

I'm currently trying to build one, fwiw.

What’s your rating, have you tried gpt4o?

It’s supposedly good up to about 1300, but aside from that the ability to prompt can make the style of play somewhat tunable for ex aggressive, defensive, etc.

Cool! I've been wondering for s while if it wouldn't be possible to use lichess games for various ratings to make typical mistakes.

I'm also curious about if it would be possible to mimic certain playing styles. Two beginners can have the same rating but one might lose because they have a weak opening, and the other one because they mess upo the end game, for example.

Random mistakes doesn't mimic human play very well.

Do you know if there are any interfaces to play against got4o? Or is it just typing in algebraic moves back and forth?

It prints an ASCII board, and then yes, you type algebraic moves back and forth and it updates the board for you each turn.

A long time ago I had the Fritz engine from chessbase. It had a sparring feature where if you maintained good play it would give up a tactical puzzle in the middle of the game. It could either warn you or not. If you didn't play solidly enough, you would just lose.

As far as I can tell, they got rid of this feature. It was the only computer opponent that felt real. Like it made a human mistake when put under pressure, rather than just playing like a computer and randomly deciding to play stupid.

Really nice work! The tabs other than "play" don't seem to be working, but I was able to try some novelty openings and it certainly felt like it was responding with human moves. It would be great to have the ability to go back/forth moves to try out different variations.

I'm curious how you combined Stockfish with your own model - but no worries if you're keeping the secret sauce a secret. All the best to you in building out this app!

I wonder if loweing the 'temperature' level (how rigid they select the max value completion) on these types of models would achieve this? Might be tricky to tune as the outcome is likely non=linear

I'm happy you enjoyed it! There are definitely a few rough edges, yes.

Since the whole thing is executed in the browser (including the model) there aren't a ton of secrets for me to keep. Essentially it is expectation maximization: the bot tries to find the move with the highest value. What is "value"? Essentially, it is the dot product between the probability distribution coming out of the model and the centipawn evaluations from Stockfish.

In other words if the model thinks you will blunder with high probability, it will try to steer you towards making that mistake.

How would you even go about making a model that can simulate a human chess skillset (saying levels implies that chess skillset is a scalar value while it is more reasonable to think of it as a tree of skills where your abilities might be higher or lower depend ending on the specific skill branch)

Hello! I built a Chess AI also named Chessmate in high school (2005) that made it into a Minecraft mod 10 years later: http://petrustheron.com/posts/chessmate.html

Java source code here: https://github.com/theronic/chessmate

take millions of games of human players of certain rating only as your learning data?

You are assuming that's going to be a reliable proxy, what would make you think that?

"Sure, you end up beating them about half the time if you cut the search down enough but it generally feels like they were still outplaying you for much of the game and you won because they made one or two blunders."

That is what winning in chess is. Minimising blunders.

> [...] feels like they were still outplaying you for much of the game and you won because they made one or two blunders.

That's why I don't like winning in multiplyer games. Usually when you win you either feel like the opponent just played comically bad on sufficient number of occasions or that they played well but in few instances you got undully lucky and it could have gone either way. Very rarely you get the desired feeling that opponent played well but you just played a little better overall so your win is deserved. It almost always seem like it's not that you are winning but the opponent is losing instead. And none of that is about AI. Making AI that lets you win symmetrical games satisfyingly and teaches you with your losses in a satisfying manner would be a billion dollar business. I don't think it can be done without some serious psychology research.

In the context of this thread (“non-GM level computer chess”, which I read as also excluding International, FIDE Master, and Candidate Master (https://en.wikipedia.org/wiki/Grandmaster_(chess))), I think it’s more important to not have a good learning algorithm.

Even 10 thousand of such games may already have way more tactics than a player at the targeted level can detect and apply. If so, a learning algorithm that detects and remembers all of them already will be better than the target level.

“The winner of a game is the one who makes the second-to-last blunder.”

(Also this has come up in computer chess; it's more important to improve the quality of your worst moves than your average moves)

condition the move on ELO while training

Exactly. Level x (whatever scalar thing the user meant by that) doesn't quite work out for the reason you outlined. X Level Players have different tactics and someone that can use all of them will likely be better than most if not all those those players. I got downvoted for saying that. Maybe I didn't phrase it as well as you did

Exactly. My eventual goal is to be able to emulate any single player with a public game history. Maybe even flag unhuman-like moves that also happen to be top stockfish moves as possible cheating.

My current chess engine already hangs its queen sometimes and walks into forks. I'm still experimenting with how to improve personalization.

Stockfish is the classic answer, though I’m not sure how well it’s graded. Someone must have a “Stockfish strength”-to-ELO mapping.

Better: take one of the existing engines, sort the moves from the best to the worst, if the top move has score S, randomly choose among the moves with score >= 0.9*S.

You can simulate a better/worse player by increasing/decreasing the factor: 1 plays as well as the chosen engine can do, 0 is typing random (yet valid) moves on the keyboard.

Yup this is better.

Any time it had an option to mate it would have a chance to mate. It's choose randomly amongst top n, not take the nth.

I don't know enough about chess but surely amateurs don't play at a consistent level. Like, I play a little and I'm quite certain some of my moves are awful (20%?) and some are the same move magnus makes (10%?) and most are on a spectrum between (70%?).

It's not getting an engine to play in the right rating range that is the problem. It's getting it to play like a human would play in that rating range.

The average rating of tournament chess players in the US is around USCF 1550. I'm not sure what their FIDE rating would be. FIDE ratings are usually 50-100 points lower than USCF ratings but that's based on comparing people that have both ratings which for the most part are strong masters and above.

A human with a USCF 1550 rating will typically be mostly making moves that are suboptimal in a variety of ways: piece coordination, king safety, planning, pawn structure, development, search, and more. Basically they are around 1550 at nearly everything. There will be variations of course. A particular player might be worse at king safety and better at tactics for instance, but it will be something like they handle king safety like a 1400 and tactics like a 1700.

With a GM level engine turned down to 1550 you tend to see aspects of GM level play still in its game. If you are a 1550 playing against it it doesn't feel like you playing the kind of opponent you will play if you enter a local chess tournament and get paired with another 1450-1650 player.

It feels like you are playing someone with a totally different approach to chess than you who just happens to lose or draw to you about the same amount as a 1450-1650 human.

It would be hilarious if they downgraded by being more aggressive. For example: It needs to score n points worth of aggressive moves that are not the best moves. After screwing up the position by n they can go back to playing the best moves again.

Otherwise you wouldn't really be learning anything useful. You would end up with an opening vocabulary that good players would easily punish. If you play crappy gambits leading to positions you know well the better players will think highly of you.

Best way to learn is to play the hardest possible engines and just take back moves when it becomes evident you've screwed up.

GPT is incapable of playing a full game of chess without making lots of illegal moves, in my experience.

Even correcting for those, it's still horrendously bad at the game. Where are you getting this 1300 number?

Yeah but, won't it also be learning from the mistakes and missed tactics too? (Assuming its reward function is telling it to predict the human's move, rather than actually trying to win.)

KataGo has a special model weights release with human-like play at various Elo: https://github.com/lightvector/KataGo/blob/master/docs/Analy...

You can see in the release notes a few screenshot examples where a particular move changes likelihood as you get to higher-level play: https://github.com/lightvector/KataGo/releases/tag/v1.15.0

What I find fasinating is how bad human beings are at chess. Now that we have engines we're finally able to analise every game ever played and they show us everything in chess that we're blind to. Their ability to never blunder in 1 or 3 moves is admirable, and better than most players, and to say nothing about their ability to make you play out the longest possible chain before checkmate. What I found most insulting is when I played against the best bot I could beat, it gave up its rook for free.

As a complete amateur I'd love a chess engine that can point out my mistakes, explain its own moves, and suggest better moves, in human terms. Like, this move would pin the white bishop. I don't know if this would be AI based or simply a search-based engine that scores in abstracted ways such as giving points for tactics like pins and forks, controlling the center, protecting the king, etc. (Typical chess search just scores based on piece and position and looks ahead farther than a human brain can handle.)

Thank you for the explanation! I completely understand the rough edges, I have some rough ideas out there myself. Would it be alright if I add a link to your site to our chess club's list of online resources?

I can also make a note of it privately and check back in with you in the future. I found it pretty remarkable that it played a human-like response to some niche openings - I actually ended up checking against Stockfish and it played different moves, which is pretty neat.

> What I want is a computer opponent that plays at a level of my choosing but plays a game that feels like that of a typical human player of that level.

Chess.com’s NPC players feel fairly in line with what you are asking for.

I recommend a chess club for that.

Sure, go ahead!

I believe that observation is borne out in the statistics too, but traditional chess training usually centers around finding the best, hard-to-find move in a position rather than avoiding blunders. I think it would be great if there was more blunder-avoidance training. In other words, a normal position where a blunder looks attractive but the player needs to avoid it.

A bit late, but I wrote up an example inference script here: https://gist.github.com/mcognetta/7a98e50859664b8efbb4ec094a...

It is a bit roundabout, since it involves converting maia models to onnx before loading into pytorch and some outdated versions of libraries (maia/lc0 are a little old). We were using this for transfer learning for a competition, so we needed some flexibility that we didn't know how to do quickly/easily in TF.

Hope this helps.

------------------

Personal note: given your interest in chess ai and your starcraft username, I think we would have a lot of shared interests. Feel free to reach out (info is in my profile).

Hi! I just messaged you.