←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
arthurcolle ◴[01 Apr 23 00:37 UTC] No.35395744[source]▶
"How much RAM did you shave off last week?"

"Oh, you know, like 12-18GB"

"Haha shut the fuck up, how much RAM did you shave off last week"

"12-18GB"

"Let me tell you what - you show me your commits right now, if you shaved off 12-18GB of RAM last week I quit my job right now and come work for you"

https://www.youtube.com/watch?v=TxHITqC5rxE

replies(1): >>35397178 #
PragmaticPulp ◴[01 Apr 23 04:35 UTC] No.35397178[source]▶
Maybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.
replies(1): >>35400418 #
lostmsu ◴[01 Apr 23 14:06 UTC] No.35400418[source]▶
It will run, just will have to reread the model for every new token.
replies(1): >>35413645 #
Szpadel ◴[02 Apr 23 18:31 UTC] No.35413645[source]▶
with nvme gen 4 ssds this might not be that huge of an issue, and for sure much cheaper than investing in ram
replies(1): >>35415612 #
lostmsu ◴[02 Apr 23 21:48 UTC] No.35415612[source]▶
I don't believe the consumer ones actually have sustained sequential read speed to saturate Gen 4.
replies(2): >>35418877 #>>35572979 #
1. Szpadel ◴[03 Apr 23 05:25 UTC] No.35418877[source]▶
sequential reads are the best case scenario for ssds. writes degrade, as they're first committed to SLC cache before being written to slower tlc/qlc.