←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
kccqzy ◴[01 Apr 23 00:36 UTC] No.35395739[source]▶
I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.
replies(5): >>35396367 #>>35396552 #>>35396848 #>>35398023 #>>35398479 #
losteric ◴[01 Apr 23 02:15 UTC] No.35396367[source]▶
Are the weights on NVME? Old SSD? HDD?
replies(2): >>35397005 #>>35402314 #
1. kccqzy ◴[01 Apr 23 17:49 UTC] No.35402314[source]▶
In my case they are on a SATA SSD.