←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.202s | source
Show context
qwertox ◴[31 Mar 23 21:36 UTC] No.35393996[source]▶
Is the 30B model clearly better than the 7B?

I played with Pi3141/alpaca-lora-7B-ggml two days ago and it was super disappointing. In percentage between 0% = alpaca-lora-7B-ggml and 100% GPT-3.5, where would LLaMA 30B be positioned?

replies(2): >>35394629 #>>35395773 #
1. Rzor ◴[31 Mar 23 22:30 UTC] No.35394629[source]▶
I haven't been able to run it myself yet, but according to what I read so far from people who did, the 30B model is where the "magic" starts to happen.