←back to thread

Auto-grading decade-old Hacker News discussions with hindsight

(karpathy.bearblog.dev)
615 points __rito__ | 6 comments | 10 Dec 25 17:23 UTC | HN request time: 0.636s | source | bottom

Related from yesterday: Show HN: Gemini Pro 3 imagines the HN front page 10 years from now - https://news.ycombinator.com/item?id=46205632
Show context
modeless ◴[10 Dec 25 19:17 UTC] No.46222213[source]▶
This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made. Some people's opinions are closer to reality than others and it's not always correlated with upvotes.

An extension of this would be to grade people on the accuracy of the comments they upvote, and use that to weight their upvotes more in ranking. I would love to read a version of HN where the only upvotes that matter are from people who agree with opinions that turn out to be correct. Of course, only HN could implement this since upvotes are private.

replies(7): >>46222748 #>>46223194 #>>46223649 #>>46224507 #>>46226548 #>>46228558 #>>46229291 #
1. TrainedMonkey ◴[10 Dec 25 20:21 UTC] No.46223194[source]▶
I long had a similar idea for stocks. Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. I would be very surprised if this had not been done a thousand times by various trading firms and enterprising individuals.

Of course in the above example of stocks there are clear predictions (HNWS will go up) and an oracle who resolves it (stock market). This seems to be a way harder problem for generic free form comments. Who resolves what prediction a particular comment has made and whether it actually happened?

replies(3): >>46223803 #>>46229280 #>>46231763 #
2. Karrot_Kream ◴[10 Dec 25 21:02 UTC] No.46223803[source]▶
I ran across Sybil [1] the other day which tries to offer a reputation score based on correct predictions in prediction markets.

[1]: https://sybilpredicttrust.info/

3. miki123211 ◴[11 Dec 25 09:20 UTC] No.46229280[source]▶
> Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy.

Didn't somebody make an ETF once that went against the prediction of some famous CNBC stock picker, showing that it would have given you alpha in the past.

> seems to be a way harder problem for generic free form comments.

That's what prediction markets are for. People for whom truth and accuracy matters (often concentrated around the rationalist community) will often very explicitly make annual lists of concrete and quantifiable predictions, and then self-grade on them later.

replies(2): >>46234106 #>>46235481 #
4. mvkel ◴[11 Dec 25 14:24 UTC] No.46231763[source]▶
Out of curiosity, I built this. I extended karpathy's code and widened the date range to see what stocks these users would pick given their sentiments.

What came back were the usual suspects: GLP-1 companies and AI.

Back to the "boring but right" thesis. Not much alpha to be found

5. Natsu ◴[11 Dec 25 17:14 UTC] No.46234106[source]▶
You probably mean Inverse Cramer:

https://finbold.com/inverse-cramer-leaves-sp-nasdaq-and-dow-...

6. red-iron-pine ◴[11 Dec 25 18:55 UTC] No.46235481[source]▶
Cramer is the stock picker guy. There is a well known "Cramer Effect" or "Cramer Bounce" where the stock peaks then drops hard.

Makes for great pump n dump if you're day trading and willing to ride

https://www.investopedia.com/terms/c/cramerbounce.asp

long-term his choices don't do well, so the Inverse Cramer basically says "do the opposite of this goober" and has solid returns (sorta; depends a lot on methodology, and the sole hedgefund playing that strategy shutdown)