Embeddings are underrated (2024)

I don't understand why some people consider the concept of many dimensions to be so mysterious. It's numbers specifying something like degrees of freedom. If I wanted to specify the position of my body sitting at my desk, I might say use two angles to specify the what is happening at each joint, and so would probably need a couple hundred to fully specify my position. A human being probably could not look at the numbers and see "he's sitting at a desk", but I don't see the conceptual difficulty".

You're right that higher dimensions are just more degrees of freedom, and mathematically it's just more numbers. But what's counterintuitive is how geometry behaves differently as dimensions grow. Things like distance, volume, and angles don’t scale the way we expect. For example, in high dimensions, almost all the volume of a sphere concentrates near its surface, and random vectors tend to be nearly orthogonal—something that rarely happens in 2D or 3D.

These effects matter in practice. In high-dimensional spaces like word embeddings, even unrelated points can seem equidistant, making basic tasks like clustering or similarity search much harder. So it's not that higher dimensions are mysterious per se, but that they defy the spatial intuitions we've developed from living in three.