←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
detrites ◴[31 Mar 23 20:58 UTC] No.35393558[source]▶
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

Ok I answered my own question.

replies(5): >>35393627 #>>35393885 #>>35393921 #>>35394786 #>>35397029 #
xienze ◴[31 Mar 23 22:46 UTC] No.35394786[source]▶
> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

It’s several things:

* Cutting-edge code, not overly concerned with optimization

* Code written by scientists, who aren’t known for being the world’s greatest programmers

* The obsession the research world has with using Python

Not surprising that there’s a lot of low-hanging fruit that can be optimized.

replies(2): >>35394897 #>>35397540 #
Miraste ◴[31 Mar 23 22:57 UTC] No.35394897[source]▶
Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
replies(3): >>35395016 #>>35395401 #>>35442394 #
chatmasta ◴[31 Mar 23 23:10 UTC] No.35395016{3}[source]▶
It might not be slow in general, but it's easy to write slow code in it.
replies(1): >>35397547 #
1. Closi ◴[01 Apr 23 05:58 UTC] No.35397547{4}[source]▶
In ML python is effectively the interface rather than the bit that is doing the heavy lifting.

The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).