Fantastic pretraining optimizers and where to find them

Haven’t read the entire paper, but the abstract seems to be focused on speed only. That is one component, but if alternative optimizers get similar results but use x2 less memory, that’s huge.