Essential Coding Theory [pdf]

Shannon is also a great way to get started understanding mathematical reasoning. Largely because he derives information entropy from basic desiderata regarding how a mathematical model of information should behave. That is, information entropy doesn’t initially have any meaning, it just fits Shannon’s requirements.

What’s brilliant about this is that Shannon accidentally arrives at a definition that is essentially identical to thermodynamic entropy (it was actually Von Neumann who pointed this out and gave it the name).

My experience is that many people fail to understand mathematics because mathematics often follows from “what do such and such rules imply” rather than building an intuitive model of the world (which is closer to where Physics traditionally falls).

Interestingly enough though, Shannon only establishes the framework which makes coding theory possible. He doesn’t actually implement any of these examples in that paper.

I used to run a book club many years ago at a startup that worked through the book version of the paper. A great way for anyone to understand both information theory specifically and mathematics in general.

Interestingly Shannon did write about entropy relating to the English language, and how given a sequence of tokens, the next token can be predicted using the probabilities of finding that token after a certain sequence in other bodies of text: http://medientheorie.com/doc/shannon_redundancy.pdf

This is from 1950. I wonder what he would have to say about today’s LLMs.