Alice's adventures in a differentiable wonderland

> Stripped of anything else, neural networks are compositions of differentiable primitives

I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence.

I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name; deep learning is simply the application of sequences of operations that are nonlinear (and hence capable of encoding arbitrary complexity) but nonetheless differentiable and so efficiently optimisable.

Before the recent AI boom, I was mystified by the possibility of AI and emulating humans (in no small part thanks to works of fiction showing AI powered androids). Then I created and trained some neural networks. Smaller ones, doing much of nothing special. That was enough to break the mysticism. To realize it was just multiplying matrices. Training them was a bit more advanced, but still applied mathematics.

Only recently have I begun to appreciate that the simplicity of the operation, applied to a large enough matrices, may still capture enough of the nature of intelligence and sentience. In the end we can be broken down into (relatively) simple chemical reactions, and it is the massive scale of these reactions that create real intelligence and sentience.