←back to thread

What Is Entropy?

(jasonfantl.com)
287 points jfantl | 5 comments | 14 Apr 25 18:32 UTC | HN request time: 0.001s | source
Show context
glial ◴[14 Apr 25 19:46 UTC] No.43685469[source]▶
One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself.

> entropy quantifies uncertainty

This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/message and their prior observations.

You and I may have different entropies about the content of the same message. If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours.

replies(4): >>43685585 #>>43686121 #>>43687411 #>>43688999 #
1. empath75 ◴[14 Apr 25 20:45 UTC] No.43686121[source]▶
> If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours.

That's got nothing to do with entropy being subjective. If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.

replies(3): >>43686265 #>>43686365 #>>43687307 #
2. glial ◴[14 Apr 25 20:59 UTC] No.43686265[source]▶
Entropy is based on your model of the world and every model, being a simplification and an estimate, is false.
3. mitthrowaway2 ◴[14 Apr 25 21:09 UTC] No.43686365[source]▶
What if I told you the dice were loaded, but I didn't tell you which face they were loaded in favor of?

Then you (presumably) assign a uniform probability over one true assumption and five false assumptions. Which is the sort of situation where subjective entropy seems quite appropriate.

4. canjobear ◴[14 Apr 25 23:13 UTC] No.43687307[source]▶
> If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.

This implies that there is an objectively true conclusion. The true probability is objective.

replies(1): >>43689052 #
5. mitthrowaway2 ◴[15 Apr 25 04:43 UTC] No.43689052[source]▶
Ok. I rolled a die and the result was 5. What should the true objective probability have been for the outcome of that roll?