Alice's adventures in a differentiable wonderland

And then you learn about binary or ternary networks where gradients don’t really exist anywhere, and you start to wonder about the importance of this differentiability.

binary networks don't really work well unless you do a relaxation first

...And then you start learning about generalizations of the notion of "gradient" to scenarios where the classical gradient doesn't exist :)