←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 2 comments | 31 Mar 23 20:37 UTC | HN request time: 0.001s | source
Show context
detrites ◴[31 Mar 23 20:58 UTC] No.35393558[source]▶
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

Ok I answered my own question.

replies(5): >>35393627 #>>35393885 #>>35393921 #>>35394786 #>>35397029 #
xienze ◴[31 Mar 23 22:46 UTC] No.35394786[source]▶
> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

It’s several things:

* Cutting-edge code, not overly concerned with optimization

* Code written by scientists, who aren’t known for being the world’s greatest programmers

* The obsession the research world has with using Python

Not surprising that there’s a lot of low-hanging fruit that can be optimized.

replies(2): >>35394897 #>>35397540 #
Miraste ◴[31 Mar 23 22:57 UTC] No.35394897[source]▶
Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
replies(3): >>35395016 #>>35395401 #>>35442394 #
1. chaorace ◴[31 Mar 23 23:54 UTC] No.35395401{3}[source]▶
Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.
replies(1): >>35432517 #
2. Max-Limelihood ◴[03 Apr 23 22:57 UTC] No.35432517[source]▶
In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.