←back to thread

Guide to Fine-Tuning LLMs

(arxiv.org)
145 points ignoramous | 2 comments | 22 Oct 24 04:41 UTC | HN request time: 0.418s | source
Show context
kcorbitt ◴[22 Oct 24 05:21 UTC] No.41911422[source]▶
I saw this when it was making the rounds on X a few days ago. Fair warning: it seems like at least some sections are AI-generated, and there isn't much insight to be gained from reading the actual sections compared to eg. reading the relevant category pages on Huggingface.
replies(4): >>41911596 #>>41911605 #>>41912253 #>>41912270 #
1. YetAnotherNick ◴[22 Oct 24 06:03 UTC] No.41911605[source]▶
Not only the it seems to be AI generated, it seems these guys don't even know about best practices or even what works. e.g. It contains archaic comparison of optimizers and its pros and cons, but for LLMs no optimizer other than Adam and new ones like Lion works.
replies(1): >>41912039 #
2. abc-1 ◴[22 Oct 24 07:32 UTC] No.41912039[source]▶
Is there a paper on this? Why do no other optimizers give good results? Adam requires insane amounts of memory so alternatives would be welcome.