←back to thread

NoProp: Training neural networks without back-propagation or forward-propagation

(arxiv.org)
161 points belleville | 3 comments | 14 Apr 25 00:03 UTC | HN request time: 0.619s | source
Show context
itsthecourier ◴[14 Apr 25 03:01 UTC] No.43677688[source]▶
"Whenever these kind of papers come out I skim it looking for where they actually do backprop.

Check the pseudo code of their algorithms.

"Update using gradient based optimizations""

replies(4): >>43677717 #>>43677878 #>>43684074 #>>43725019 #
f_devd ◴[14 Apr 25 03:47 UTC] No.43677878[source]▶
I mean the only claim is no propagation, you always need a gradient of sorts to update parameters. Unless you just stumble upon the desired parameters. Even genetic algorithms effectively has gradients which are obfuscated through random projections.
replies(3): >>43678034 #>>43679597 #>>43679675 #
erikerikson ◴[14 Apr 25 04:24 UTC] No.43678034[source]▶
No you don't. See Hebbian learning (neurons that fire together wire together). Bonus: it is one of the biologically plausible options.

Maybe you have a way of seeing it differently so that this looks like a gradient? Gradient keys my brain into a desired outcome expressed as an expectation function.

replies(4): >>43678091 #>>43679021 #>>43680033 #>>43683591 #
1. red75prime ◴[14 Apr 25 04:36 UTC] No.43678091[source]▶
> See Hebbian learning

The one that is not used, because it's inherently unstable?

Learning using locally accessible information is an interesting approach, but it needs to be more complex than "fire together, wire together". And then you might have propagation of information that allows to approximate gradients locally.

replies(1): >>43678117 #
2. erikerikson ◴[14 Apr 25 04:43 UTC] No.43678117[source]▶
Is that what they're teaching now? Originally it was not used because it was believed it couldn't learn XOR (it can [just not as perceptrons were defined]).

Is there anyone in particular whose work focuses on this that you know of?

replies(1): >>43679247 #
3. ckcheng ◴[14 Apr 25 08:31 UTC] No.43679247[source]▶
Oja's rule dates back to 1982?

It’s Hebbian and solves all stability problems.

https://en.wikipedia.org/wiki/Oja's_rule