Alice's adventures in a differentiable wonderland

It’s not ‘inaccurate’. The mark of true mastery is an ability to make terse statements that convey a huge amount without involving excessive formality or discussion of by-the-by technical details. If ever you’ve spoken to world-renowned experts in pure mathematics or other highly technical and pendantic fields, you’ll find they’ll say all sorts of ‘inaccurate’ things in conversation (or even in written documents). It doesn’t make them worthless; far from it.

If you want to have a war of petty pedantry, let’s go: the derivative of ReLU can’t be discontinuous at zero, as you say, because continuity (or indeed discontinuity) of a function at x requires the function to have a value at x (which is the negation of what your first statement correctly claims).

my experience with world-renowned experts in pure mathematics is that they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate, because their discipline requires them to be very clear about precisely what they are saying

discontinuity of a function at x does not, according to the usual definition of 'continuity', require the function to have a value at x; indeed, functions that fail to have a value at x are necessarily discontinuous there, precisely because (as you say) they are not continuous there. https://en.wikipedia.org/wiki/Continuous_function#Definition...

there are other definitions of 'discontinuous' in use, but i can't think of one that would give the result you claim

> they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate

Sure. But what part of this entirely worded in natural language, and very short statement made you think it was a technical, formal statement? I think you’re just taking an opportunity to flex your knowledge of basic calculus, and deliberately attributing intent to the author that isn’t there in order to look clever.

Regarding a function being discontinuous at a point outside its domain: if you take a completely naive view of what ‘discontinuous’ means, then I suppose you can say so. But discontinuity is just the logical negation of continuity. Observe:

To say that f: X —> Y (in this context, a real-valued function of real numbers) is continuous means precisely

∀x∈X ∀ε>0 ∃δ>0 |x - p| < δ ⇒ |f(x) - f(p)| < ε

and so its negation looks like

∃x∈X ⌐ …

that is, there is a point in X, the domain of f where continuity fails.

For example, you wouldn’t talk about a function defined on the integers being discontinuous at pi, would you? That would just be weird.

To prove the point further, observe that the set of discontinuities (according to your definition) of any given function would actually include every number… in fact every mathematical object in the universe — which would make it not even a set in ZFC. So it’s absurd.

Even more reasons to believe functions can only be discontinuous at points of their domain: a function is said to be discontinuous if it has at least one discontinuity. By your definition, every function is discontinuous.

…anyway, I said we were going to be petty. I’m trying to demonstrate this is a waste of time by wasting my own time.

Because memes aren't allowed on HN, you're not allowed to reply with the "akssshuallllyyy" meme, so you had to go to these lengths.

¯\_(ツ)_/¯

You’re actually not far off. I’m somewhat embarrassed by the above, but I think it makes the point.

Invoking excessive formality and discussions of minute technical details leads to a cathedral of knowledge built on autistic pedantry. The chosen rabbit hole to get lost in needs to be the correct one. And human science is riddled with the paths that have naive or childish fundamentals.

This comment makes me want to both upvote and downvote with extreme enthusiasm/fury!

The sign of a truly good conversation?

Lack of differentiability is actually a very important feature of the underlying optimization problem.

You might think that it doesn't matter because ReLU is, e.g., non-differentiable "only at one point".

Gradient based methods (what you find in pytorch) generally rely on the idea that gradients should taper to 0 in the proximity of a local optimum. This is not the case for non-differentiable functions, and in fact gradients can be made to be arbitrarily large even very close to the optimum.

As you may imagine, it is not hard to construct examples where simple gradient methods that do not properly take these facts into account fail to converge. These examples are not exotic.

> a cathedral of knowledge built on autistic pedantry

this is certainly true, but more often we use its short name, 'math'. it turns out to be far more effective than so-called common sense

you have an interesting point of view, and some of the things you have said are correct, but if you try to use gradient descent on a function from, say, ℤ → ℝ, you are going to be a very sad xanda. i would indeed describe such a function as being discontinuous not just at π but everywhere, at least with the usual definition of continuity (though there is a sense in which such a function could be, for example, scott-continuous)

even in the case of a single discontinuity in the derivative, like in relu', you lose the intermediate value theorem and everything that follows from it; it's not an inconsequential or marginally relevant fact

Note that any function ℤ → ℝ is continuous on its domain but nowhere differentiable.

A Scott-continuous function ℤ → ℝ must be monontonous. So not every such function is Scott-continuous.

aha, thanks!