Auto-grading decade-old Hacker News discussions with hindsight

This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made. Some people's opinions are closer to reality than others and it's not always correlated with upvotes.

An extension of this would be to grade people on the accuracy of the comments they upvote, and use that to weight their upvotes more in ranking. I would love to read a version of HN where the only upvotes that matter are from people who agree with opinions that turn out to be correct. Of course, only HN could implement this since upvotes are private.

The RES (Reddit Enhancement Suite) browser extension indirectly does this for me since it tracks the lifetime number of upvotes I give other users. So when I stumble upon a thread with a user with like +40 I know "This is someone whom I've repeatedly found to have good takes" (depending on the context).

It's subjective of course but at least it's transparently so.

I just think it's neat that it's kinda sorta a loose proxy for what you're talking about but done in arguably the simplest way possible.

I am not a Redditor, but RES sounds like it would increase the ‘echo-chamber’ effect, rather than improving one’s understanding of contributors’ calibration.

I long had a similar idea for stocks. Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. I would be very surprised if this had not been done a thousand times by various trading firms and enterprising individuals.

Of course in the above example of stocks there are clear predictions (HNWS will go up) and an oracle who resolves it (stock market). This seems to be a way harder problem for generic free form comments. Who resolves what prediction a particular comment has made and whether it actually happened?

it depends on if you vote based on the quality of contribution to the discussion or based on how much you agree/disagree.

Reddit's current structure very much produces an echo chamber with only one main prevailing view. If everyone used an extension like this I would expect it to increase overall diversity of opinion on the site, as things that conflict with the main echo chamber view could still thrive in their own communities rather than getting downvoted with the actual spam.

More than having exact same system but with any random reader voting ? I'd say as long as you don't do "I disagree therefore I downvote" it would probably be more accurate than having essentially same voting system driven by randoms like reddit/HN already does

The problem seems underspecified; what does it mean for a comment to be accurate? It would seem that comments like "the sun will rise tomorrow" would rank highest, but they aren't surprising.

I ran across Sybil [1] the other day which tries to offer a reputation score based on correct predictions in prediction markets.

[1]: https://sybilpredicttrust.info/

That assumes your upvotes in the past were a good proxy for being correct today. You could have both been wrong.

That’s what Elon’s vision was before he ended up buying Twitter. Keep a digital track record for journalists. He wanted to call it Pravda.

(And we do have that in real life. Just as, among friends, we do keep track of who is in whose debt, we also keep a mental map of whose voice we listen to. Old school journalism still had that, where people would be reading someone’s column over the course of decades. On the internet, we don’t have that, or we have it rarely.)

>This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made.

Why stop there?

If you can do that you can score them on all sorts of things. You could make a "this person has no moral convictions and says whatever makes the number go up" score. Or some other kind of score.

Stuff like this makes the community "smaller" in a way. Like back in the old days on forums and IRC you knew who the jerks were.

Didn't Slashdot have something like the second point with their meta-moderation, many many years ago?

Echo chambers will always result on social media. I don't think you can come up with a format that will not result in consolidated blocs.

I don't think you can change user behavior like this.

You can give them a "venting sink" though. Instead of having a downvote button that just downvotes, have it pop up a little menu asking for a downvote reason, with "spam" and "disagree" as options. You could then weigh downvotes by which option was selected, along with an algorithm to discover "user honesty" based on whether their downvotes correlate with others or just with the people on their end of the political spectrum, a la Birdwatch.

> Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy.

Didn't somebody make an ETF once that went against the prediction of some famous CNBC stock picker, showing that it would have given you alpha in the past.

> seems to be a way harder problem for generic free form comments.

That's what prediction markets are for. People for whom truth and accuracy matters (often concentrated around the rationalist community) will often very explicitly make annual lists of concrete and quantifiable predictions, and then self-grade on them later.

I like the idea and certainly would try it. Although I feel in a way this would be an anti-thesis to HN. HN tries to foster curiosity, but if you're (only) ranked by the accuracy of your predictions, this could give the incentive to always fall back to a save and boring position.

Echo chamber of rational, thoughtful and truthful speakers is what I’m looking for in Internet forums.

just because an idea is qualitative doesn't mean its invalid

Out of curiosity, I built this. I extended karpathy's code and widened the date range to see what stocks these users would pick given their sentiments.

What came back were the usual suspects: GLP-1 companies and AI.

Back to the "boring but right" thesis. Not much alpha to be found

You probably mean Inverse Cramer:

https://finbold.com/inverse-cramer-leaves-sp-nasdaq-and-dow-...

That’s what everyone living in an echo chamber (and especially one of their own creation) thinks they’re in.

flat earth creationists would describe their colleagues the same way.

a group of them certainly is an echo chamber; why isn't your view?

Cramer is the stock picker guy. There is a well known "Cramer Effect" or "Cramer Bounce" where the stock peaks then drops hard.

Makes for great pump n dump if you're day trading and willing to ride

https://www.investopedia.com/terms/c/cramerbounce.asp

long-term his choices don't do well, so the Inverse Cramer basically says "do the opposite of this goober" and has solid returns (sorta; depends a lot on methodology, and the sole hedgefund playing that strategy shutdown)

I don't think I'm in any is my problem (HN is better than most, doesn't mean it's good in absolute terms...)

"you're in an echo chamber" is one of the most frightfully overused opinions.

Hacker News structure is identical though. Topics invite different demographics and downvotes suppress unpopular opinions. The front page shows most up voted stories. It's the same system.

The expression is an echo chamber in and of itself; it is self-fulfilling prophecy.

Sorta.

IIRC, when comment moderation and scoring came to Slashdot, only a random (and changing) selection of users were able to moderate.

Meta-moderation came a bit later. It allowed people to review prior moderation actions and evaluate the worth of those actions.

Those users who made good moderations were more likely to become a mod again in the future than those who made bad moderations.

The meta-mods had no idea whose actions they were evaluating, and previous/potential mods had no idea what their score was. That anonymity helped keep it honest and harder to game.

HN's moderation and ranking is better. But there's definitely an echo chamber effect here too.

He doesn't deny that his point of view forms an echo chamber.

An echo chamber is a product of your own creation. If you're willing to upvote people who disagree with your and actively seek out opposite takes to be genuinely curious about, then you're probably not in an echo chamber.

The tools for controlling your feed are reducing on social media like Instagram, TikTok, Youtube, etc., but simply saying that you follow and respect the opinions of a select group doesn't necessarily mean you're forming an echo chamber.

This is different from something like flat earth/other conspiracy theories where when confronted with opposite evidence, they aren't likely to engage with it in good faith.

You can't change it for other users, only for yourself, which is what the original comment about the extension said.

HN has some built-in ways to reduce this, like not allowing everyone to downvote everything.