←back to thread

Backpropagation is a leaky abstraction (2016)

(karpathy.medium.com)
346 points swatson741 | 1 comments | 02 Nov 25 05:20 UTC | HN request time: 0.207s | source
Show context
brcmthrowaway ◴[02 Nov 25 08:45 UTC] No.45788805[source]▶
Do LLMs still use backprop?
replies(3): >>45788887 #>>45790627 #>>45791775 #
1. samsartor ◴[02 Nov 25 17:10 UTC] No.45791775[source]▶
Yes. Pretraining and fine-tuning use standard Adam optimizers (usually with weight-decay). Reinforcement learning has been the odd-man out historically, but these days almost all RL algorithms also use backprop and gradient descent.