←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
detrites ◴[31 Mar 23 20:58 UTC] No.35393558[source]▶
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

Ok I answered my own question.

replies(5): >>35393627 #>>35393885 #>>35393921 #>>35394786 #>>35397029 #
datadeft ◴[31 Mar 23 21:04 UTC] No.35393627[source]▶
I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
replies(2): >>35393655 #>>35393662 #
terafo ◴[31 Mar 23 21:06 UTC] No.35393655[source]▶
You mean in contained app? It can already run on a phone. GPU acceleration would be nice at this point, though.
replies(1): >>35407867 #
1. datadeft ◴[02 Apr 23 06:52 UTC] No.35407867[source]▶
I was thinking about the limitation by the mobile HW. Yeah GPU support would be nice.