←back to thread

Guide to Fine-Tuning LLMs

(arxiv.org)
145 points ignoramous | 1 comments | 22 Oct 24 04:41 UTC | HN request time: 0.203s | source
Show context
kcorbitt ◴[22 Oct 24 05:21 UTC] No.41911422[source]▶
I saw this when it was making the rounds on X a few days ago. Fair warning: it seems like at least some sections are AI-generated, and there isn't much insight to be gained from reading the actual sections compared to eg. reading the relevant category pages on Huggingface.
replies(4): >>41911596 #>>41911605 #>>41912253 #>>41912270 #
1. danielhanchen ◴[22 Oct 24 08:17 UTC] No.41912270[source]▶
I took a skim through it in the morning - I like the LoRA Learns Less and Forgets Less paper more https://openreview.net/forum?id=aloEru2qCG - it has much more signal in a few pages - also the original QLoRA paper from Dettmers https://arxiv.org/abs/2305.14314 has so many more important morsels.

But all in all, the review is a reasonable "manual" I guess. I would have liked maybe more instructive comprehensive practical examples, and maybe more mention of other OSS packages for finetuning :))