←back to thread

Softmax forever, or why I like softmax

(kyunghyuncho.me)
181 points jxmorris12 | 1 comments | 16 Feb 25 07:08 UTC | HN request time: 0s | source
Show context
stared ◴[20 Feb 25 10:07 UTC] No.43113011[source]▶
There are many useful tricks - like cosine distance.

In contrast, softmax has a very deep grounding in statistical physics - where it is called the Boltzmann distribution. In fact, this connection between statistical physics and machine learning was so fundamental that it was a key part of the 2024 Nobel Prize in Physics awarded to Hopfield and Hinton.

replies(1): >>43114597 #
LudwigNagasena ◴[20 Feb 25 13:49 UTC] No.43114597[source]▶
Study of thermodynamics gave rise to many concepts in information theory and statistics, but I wouldn't say that there is any direct connection per se between thermodynamics and any field where statistics or information theory are applicable. And the reasoning behind the 2024 Nobel Prize in Physics was... quite innovative.
replies(1): >>43115298 #
1. mitthrowaway2 ◴[20 Feb 25 14:47 UTC] No.43115298[source]▶
> I wouldn't say that there is any direct connection per se between thermodynamics and any field where statistics or information theory are applicable.

Thermodynamics can absolutely be studied through both a statistical mechanics and an information theory lens, and many physicists have found this to be quite productive and enlightening. Especially when it gets to tricky cases involving entropy, like Maxwell's Demon and Landauer's Eraser, one struggles not to do so.