Alice's adventures in a differentiable wonderland

> Stripped of anything else, neural networks are compositions of differentiable primitives

I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence.

I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name; deep learning is simply the application of sequences of operations that are nonlinear (and hence capable of encoding arbitrary complexity) but nonetheless differentiable and so efficiently optimisable.

I feel like this statement is both obvious after spending a few minutes working with neural networks and completely useless in helping you build better neural networks.

It's kind of like saying, "Stripped of anything else, works of literature are compositions of words"