←back to thread

161 points belleville | 4 comments | | HN request time: 1.003s | source
Show context
itsthecourier ◴[] No.43677688[source]
"Whenever these kind of papers come out I skim it looking for where they actually do backprop.

Check the pseudo code of their algorithms.

"Update using gradient based optimizations""

replies(4): >>43677717 #>>43677878 #>>43684074 #>>43725019 #
f_devd ◴[] No.43677878[source]
I mean the only claim is no propagation, you always need a gradient of sorts to update parameters. Unless you just stumble upon the desired parameters. Even genetic algorithms effectively has gradients which are obfuscated through random projections.
replies(3): >>43678034 #>>43679597 #>>43679675 #
erikerikson ◴[] No.43678034[source]
No you don't. See Hebbian learning (neurons that fire together wire together). Bonus: it is one of the biologically plausible options.

Maybe you have a way of seeing it differently so that this looks like a gradient? Gradient keys my brain into a desired outcome expressed as an expectation function.

replies(4): >>43678091 #>>43679021 #>>43680033 #>>43683591 #
1. yobbo ◴[] No.43679021[source]
If there is a weight update, there is a gradient, and a loss objective. You might not write them down explicitly.

I can't recall exactly what the Hebbian update is, but something tells me it minimises the "reconstruction loss", and effectively learns the PCA matrix.

replies(2): >>43680272 #>>43682329 #
2. orbifold ◴[] No.43680272[source]
Not every vector field has a potential. So not every weight update can be written as a gradient.
replies(1): >>43682930 #
3. erikerikson ◴[] No.43682329[source]
> loss objective

There is no prediction or desired output, certainly explicit. I was playing with those things in my work to try and understand how our brains cause the emergence of intelligence rather than solve some classification or related problem. What I managed to replicate was the learning of XOR by some nodes and further that multidimensional XORs up to the number of inputs could be learned.

Perhaps you can say that PCAish is the implicit objective/result but I still reject that there is any conceptual notion of what a node "should" output even if iteratively applying the learning rule leads us there.

4. yobbo ◴[] No.43682930[source]
True.