←back to thread

Something weird is happening with LLMs and chess

(dynomight.substack.com)
695 points crescit_eundo | 3 comments | 14 Nov 24 17:05 UTC | HN request time: 0.453s | source
Show context
niobe ◴[15 Nov 24 00:40 UTC] No.42142885[source]▶
I don't understand why educated people expect that an LLM would be able to play chess at a decent level.

It has no idea about the quality of it's data. "Act like x" prompts are no substitute for actual reasoning and deterministic computation which clearly chess requires.

replies(20): >>42142963 #>>42143021 #>>42143024 #>>42143060 #>>42143136 #>>42143208 #>>42143253 #>>42143349 #>>42143949 #>>42144041 #>>42144146 #>>42144448 #>>42144487 #>>42144490 #>>42144558 #>>42144621 #>>42145171 #>>42145383 #>>42146513 #>>42147230 #
mannykannot ◴[15 Nov 24 04:47 UTC] No.42144041[source]▶
One of the main purposes of running experiments of any sort is to find out if our preconceptions are accurate. Of course, if someone is not interested in that question, they might as well choose not to look through the telescope.
replies(1): >>42144411 #
bowsamic ◴[15 Nov 24 06:31 UTC] No.42144411[source]▶
Sadly there’s a common sentiment on HN that testing obvious assumptions is a waste of time
replies(2): >>42144775 #>>42145625 #
1. BlindEyeHalo ◴[15 Nov 24 07:55 UTC] No.42144775[source]▶
Not only on HN. Trying to publish a scientific article that does not contain the word 'novel' has become almost impossible. No one is trying to reproduce anyones claims anymore.
replies(2): >>42145089 #>>42145377 #
2. pcf ◴[15 Nov 24 08:57 UTC] No.42145089[source]▶
Do you think this bias is part of the replication crisis in science?
3. bowsamic ◴[15 Nov 24 09:50 UTC] No.42145377[source]▶
I don't think this is about replication, but even just about the initial test in the first place. In science we do often test obvious things. For example, I was a theoretical quantum physicist, and a lot of the time I knew that what I am working on will definitely work, since the maths checks out. In some sense that makes it kinda obvious, but we test it anyway.

The issue is that even that kinda obviousness is criticised here. People get mad at the idea of doing experiments when we already expect a result.