←back to thread

Auto-grading decade-old Hacker News discussions with hindsight

(karpathy.bearblog.dev)
615 points __rito__ | 2 comments | 10 Dec 25 17:23 UTC | HN request time: 0.001s | source

Related from yesterday: Show HN: Gemini Pro 3 imagines the HN front page 10 years from now - https://news.ycombinator.com/item?id=46205632
Show context
hackthemack ◴[10 Dec 25 19:36 UTC] No.46222507[source]▶
I noticed the Hall of Fame grading of predictive comments has a quirk? It grades some comments about if they came true or not, but in the grading of comment to the article

https://news.ycombinator.com/item?id=10654216

The Cannons on the B-29 Bomber "accurate account of LeMay stripping turrets and shifting to incendiary area bombing; matches mainstream history"

It gave a good grade to user cstross but to my reading of the comment, cstross just recounted a bit of old history. The evaluation gave cstross for just giving a history lesson or no?

replies(1): >>46222634 #
1. karpathy ◴[10 Dec 25 19:44 UTC] No.46222634[source]▶
Yes I noticed a few of these around. The LLM is a little too willing to give out grades for comments that were good/bad in a bit more general sense, even if they weren't making strong predictions specifically. Another thing I noticed is that the LLM has a very impressive recognition of the various usernames and who they belong to, and I think shows a little bit of a bias in its evaluations based on the identity of the person. I tuned the prompt a little bit based on some low-hanging fruit mistakes but I think one can most likely iterate it quite a bit further.
replies(1): >>46227849 #
2. patcon ◴[11 Dec 25 05:00 UTC] No.46227849[source]▶
I think you were getting at this, but in case others didn't know: cstross is a famous sci-fi author and futurist :)