←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 3 comments | 31 Mar 23 20:37 UTC | HN request time: 0.407s | source
Show context
arthurcolle ◴[01 Apr 23 00:37 UTC] No.35395744[source]▶
"How much RAM did you shave off last week?"

"Oh, you know, like 12-18GB"

"Haha shut the fuck up, how much RAM did you shave off last week"

"12-18GB"

"Let me tell you what - you show me your commits right now, if you shaved off 12-18GB of RAM last week I quit my job right now and come work for you"

https://www.youtube.com/watch?v=TxHITqC5rxE

replies(1): >>35397178 #
PragmaticPulp ◴[01 Apr 23 04:35 UTC] No.35397178[source]▶
Maybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.
replies(1): >>35400418 #
lostmsu ◴[01 Apr 23 14:06 UTC] No.35400418[source]▶
It will run, just will have to reread the model for every new token.
replies(1): >>35413645 #
Szpadel ◴[02 Apr 23 18:31 UTC] No.35413645[source]▶
with nvme gen 4 ssds this might not be that huge of an issue, and for sure much cheaper than investing in ram
replies(1): >>35415612 #
1. lostmsu ◴[02 Apr 23 21:48 UTC] No.35415612[source]▶
I don't believe the consumer ones actually have sustained sequential read speed to saturate Gen 4.
replies(2): >>35418877 #>>35572979 #
2. Szpadel ◴[03 Apr 23 05:25 UTC] No.35418877[source]▶
sequential reads are the best case scenario for ssds. writes degrade, as they're first committed to SLC cache before being written to slower tlc/qlc.
3. Robotbeat ◴[14 Apr 23 18:25 UTC] No.35572979[source]▶
Gen 5 pcie is ~4GB/s per lane, AMD Genoa chips have 128 such lanes. That means on the order of 500GB/s aggregate throughput, which is comparable to the aggregate theoretical throughput of the 12 channel DDR5 RAM of the Genoa CPUs.

In other words, with enough data interleaving between enough NVME SSDs, you should have SSD throughput of the same order of magnitude as the system RAM.

The weights are static, so it’s just reads.