NoProp: Training neural networks without back-propagation or forward-propagation

No you don't. See Hebbian learning (neurons that fire together wire together). Bonus: it is one of the biologically plausible options.

Maybe you have a way of seeing it differently so that this looks like a gradient? Gradient keys my brain into a desired outcome expressed as an expectation function.

> See Hebbian learning

The one that is not used, because it's inherently unstable?

Learning using locally accessible information is an interesting approach, but it needs to be more complex than "fire together, wire together". And then you might have propagation of information that allows to approximate gradients locally.

Is that what they're teaching now? Originally it was not used because it was believed it couldn't learn XOR (it can [just not as perceptrons were defined]).

Is there anyone in particular whose work focuses on this that you know of?

If there is a weight update, there is a gradient, and a loss objective. You might not write them down explicitly.

I can't recall exactly what the Hebbian update is, but something tells me it minimises the "reconstruction loss", and effectively learns the PCA matrix.

Oja's rule dates back to 1982?

It’s Hebbian and solves all stability problems.

https://en.wikipedia.org/wiki/Oja's_rule

Even with Hebbian learning, isn't there a synapse strength? If so, then you at least need a direction (+/-) if not a specific gradient value.

Not every vector field has a potential. So not every weight update can be written as a gradient.

Yes there is a weight on every connection. At least when I was at it gradients were talked about in reference to the solution space (e.g. gradient descent). The implication is that there is some notion of what is "correct"for some neutron to have output and then we bend it to our will by updating the weight. In Hebbian learning there isn't a notion of correct activation, just a calculation over the local environment.

> loss objective

There is no prediction or desired output, certainly explicit. I was playing with those things in my work to try and understand how our brains cause the emergence of intelligence rather than solve some classification or related problem. What I managed to replicate was the learning of XOR by some nodes and further that multidimensional XORs up to the number of inputs could be learned.

Perhaps you can say that PCAish is the implicit objective/result but I still reject that there is any conceptual notion of what a node "should" output even if iteratively applying the learning rule leads us there.

True.

Nope that update with the rank one update is exactly the projected gradient of the reconstruction loss. That's not the way it is usually taught. So Hebbian learning was an unfortunate example.

Gradient descent is only one way of searching for a minima, so in that sense it is not necessary, for example, when one can analytically solve for the extrema of the loss. As an alternative one could do Monte Carlo search instead of gradient descent. For a convex loss that would be less efficient of course.