←back to thread

Something weird is happening with LLMs and chess

(dynomight.substack.com)
688 points crescit_eundo | 9 comments | 14 Nov 24 17:05 UTC | HN request time: 0s | source | bottom
Show context
swiftcoder ◴[15 Nov 24 07:57 UTC] No.42144784[source]▶
I feel like the article neglects one obvious possibility: that OpenAI decided that chess was a benchmark worth "winning", special-cases chess within gpt-3.5-turbo-instruct, and then neglected to add that special-case to follow-up models since it wasn't generating sustained press coverage.
replies(8): >>42145306 #>>42145352 #>>42145619 #>>42145811 #>>42145883 #>>42146777 #>>42148148 #>>42151081 #
scott_w ◴[15 Nov 24 11:10 UTC] No.42145811[source]▶
I suspect the same thing. Rather than LLMs “learning to play chess,” they “learnt” to recognise a chess game and hand over instructions to a chess engine. If that’s the case, I don’t feel impressed at all.
replies(5): >>42146086 #>>42146152 #>>42146383 #>>42146415 #>>42156785 #
1. antifa ◴[15 Nov 24 12:47 UTC] No.42146415[source]▶
TBH I think a good AI would have access to a Swiss army knife of tools and know how to use them. For example a complicated math equation, using a calculator is just smarter than doing it in your head.
replies(1): >>42146582 #
2. PittleyDunkin ◴[15 Nov 24 13:11 UTC] No.42146582[source]▶
We already have the chess "calculator", though. It's called stockfish. I don't know why you'd ask a dictionary how to solve a math problem.
replies(4): >>42146684 #>>42147106 #>>42149986 #>>42162440 #
3. iamacyborg ◴[15 Nov 24 13:23 UTC] No.42146684[source]▶
People ask LLM’s to do all sorts of things they’re not good at.
4. the_af ◴[15 Nov 24 14:12 UTC] No.42147106[source]▶
A generalist AI with a "chatty" interface that delegates to specialized modules for specific problem-solving seems like a good system to me.

"It looks like you're writing a letter" ;)

replies(1): >>42147436 #
5. datadrivenangel ◴[15 Nov 24 14:54 UTC] No.42147436{3}[source]▶
Lets clip this in the bud before it grows wings.
replies(1): >>42150584 #
6. mkipper ◴[15 Nov 24 19:23 UTC] No.42149986[source]▶
Chess might not be a great example, given that most people interested in analyzing chess moves probably know that chess engines exist. But it's easy to find examples where this approach would be very helpful.

If I'm an undergrad doing a math assignment and want to check an answer, I may have no idea that symbolic algebra tools exist or how to use them. But if an all-purpose LLM gets a screenshot of a math equation and knows that its best option is to pass it along to one of those tools, that's valuable to me even if it isn't valuable to a mathematician who would have just cut out of the LLM middle-man and gone straight to the solver.

There are probably a billion examples like this. I'd imagine lots of people are clueless that software exists which can help them with some problem they have, so an LLM would be helpful for discovery even if it's just acting as a pass-through.

replies(1): >>42151710 #
7. nuancebydefault ◴[15 Nov 24 20:27 UTC] No.42150584{4}[source]▶
It looks like you have a deja vu
8. mabster ◴[15 Nov 24 22:04 UTC] No.42151710{3}[source]▶
Even knowing that the software exists isn't enough. You have to learn how to use the thing.
9. threatripper ◴[17 Nov 24 06:53 UTC] No.42162440[source]▶
You take a picture of a chess board and send it to ChatGPT and it replies with the current evaluation and the best move/strategy for black and white.