←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 2 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
abujazar ◴[31 Mar 23 22:31 UTC] No.35394638[source]▶
I love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.
replies(2): >>35395088 #>>35398259 #
faitswulff ◴[31 Mar 23 23:18 UTC] No.35395088[source]▶
How so?
replies(2): >>35395298 #>>35399707 #
kccqzy ◴[31 Mar 23 23:43 UTC] No.35395298[source]▶
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things.

This is totally the right way. Make it work, then make it right, then make it fast.

replies(5): >>35395814 #>>35395908 #>>35396260 #>>35396337 #>>35397136 #
buzzert ◴[01 Apr 23 01:03 UTC] No.35395908[source]▶
Funny enough, this tracks the early history of Google as well. It was originally written by Larry Page and Sergey Brin (both grad students at the time) in Python, then Sanjay Ghemawat rewrote the whole thing in C++.
replies(1): >>35415209 #
1. heap_perms ◴[02 Apr 23 21:07 UTC] No.35415209{3}[source]▶
That sounds very interesting. Anyone know where I can find more on this story?
replies(1): >>35430251 #
2. buzzert ◴[03 Apr 23 19:50 UTC] No.35430251[source]▶
I learned this from Steven Levy's great book, "In the Plex": https://www.amazon.com/Plex-Google-Thinks-Works-Shapes/dp/14...