←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.212s | source
Show context
brucethemoose2 ◴[31 Mar 23 20:46 UTC] No.35393393[source]▶
Does that also mean 6GB VRAM?

And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4

replies(2): >>35393441 #>>35393450 #
1. terafo ◴[31 Mar 23 20:49 UTC] No.35393441[source]▶
No(llama.cpp is cpu-only) and no(you need to requantize the model).