←back to thread

Something weird is happening with LLMs and chess

(dynomight.substack.com)
688 points crescit_eundo | 3 comments | 14 Nov 24 17:05 UTC | HN request time: 0.626s | source
Show context
swiftcoder ◴[15 Nov 24 07:57 UTC] No.42144784[source]▶
I feel like the article neglects one obvious possibility: that OpenAI decided that chess was a benchmark worth "winning", special-cases chess within gpt-3.5-turbo-instruct, and then neglected to add that special-case to follow-up models since it wasn't generating sustained press coverage.
replies(8): >>42145306 #>>42145352 #>>42145619 #>>42145811 #>>42145883 #>>42146777 #>>42148148 #>>42151081 #
scott_w ◴[15 Nov 24 11:10 UTC] No.42145811[source]▶
I suspect the same thing. Rather than LLMs “learning to play chess,” they “learnt” to recognise a chess game and hand over instructions to a chess engine. If that’s the case, I don’t feel impressed at all.
replies(5): >>42146086 #>>42146152 #>>42146383 #>>42146415 #>>42156785 #
fires10 ◴[15 Nov 24 11:52 UTC] No.42146086[source]▶
Recognize and hand over to a specialist engine? That might be useful for AI. Maybe I am missing something.
replies(5): >>42146145 #>>42146293 #>>42146329 #>>42147558 #>>42151536 #
1. generic92034 ◴[15 Nov 24 12:30 UTC] No.42146293[source]▶
How do you think AI are (correctly) solving simple mathematical questions which they have not trained for directly? They hand it over to a specialist maths engine.
replies(1): >>42149781 #
2. internetter ◴[15 Nov 24 19:00 UTC] No.42149781[source]▶
This is a relatively recent development (<3 months), at least for OpenAI, where the model will generate code to solve math and use the response
replies(1): >>42151065 #
3. cruffle_duffle ◴[15 Nov 24 21:07 UTC] No.42151065[source]▶
They’ve been doing that a lot longer than three months. ChatGPT has been handing stuff off to python for a very long time. At least for my paid account anyway.