←back to thread

NoProp: Training neural networks without back-propagation or forward-propagation

(arxiv.org)
161 points belleville | 1 comments | 14 Apr 25 00:03 UTC | HN request time: 0.217s | source
Show context
itsthecourier ◴[14 Apr 25 03:01 UTC] No.43677688[source]▶
"Whenever these kind of papers come out I skim it looking for where they actually do backprop.

Check the pseudo code of their algorithms.

"Update using gradient based optimizations""

replies(4): >>43677717 #>>43677878 #>>43684074 #>>43725019 #
1. scarmig ◴[14 Apr 25 17:48 UTC] No.43684074[source]▶
Check out feedback alignment. You provide feedback with a random static linear transformation of the loss to earlier layers, and they eventually align with the feedback matrix to enable learning.

It's certifiably insane that it works at all. And not even vaguely backprop, though if you really wanted to stretch the definition I guess you could say that the feedforward layers align to take advantage of a synthetic gradient in a way that approximates backprop.