Most active commenters
  • faitswulff(3)

←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 15 comments | 31 Mar 23 20:37 UTC | HN request time: 2.126s | source | bottom
Show context
abujazar ◴[31 Mar 23 22:31 UTC] No.35394638[source]▶
I love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.
replies(2): >>35395088 #>>35398259 #
1. faitswulff ◴[31 Mar 23 23:18 UTC] No.35395088[source]▶
How so?
replies(2): >>35395298 #>>35399707 #
2. kccqzy ◴[31 Mar 23 23:43 UTC] No.35395298[source]▶
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things.

This is totally the right way. Make it work, then make it right, then make it fast.

replies(5): >>35395814 #>>35395908 #>>35396260 #>>35396337 #>>35397136 #
3. Hortinstein ◴[01 Apr 23 00:49 UTC] No.35395814[source]▶
Not sure if you came up with "Make it work, then make it right, then make it fast." but I just screenshotted it and made it the mantra for my current project...which is by far more complicated than anything I have done as a side project. I am struggling with the desire to go "make it right" as I work on shipping the deployable prototype (right now running on cloud services I control)...thanks for this
replies(1): >>35396299 #
4. buzzert ◴[01 Apr 23 01:03 UTC] No.35395908[source]▶
Funny enough, this tracks the early history of Google as well. It was originally written by Larry Page and Sergey Brin (both grad students at the time) in Python, then Sanjay Ghemawat rewrote the whole thing in C++.
replies(1): >>35415209 #
5. faitswulff ◴[01 Apr 23 01:57 UTC] No.35396260[source]▶
I meant "how is the Python mess getting cleaned up," rather.
6. faitswulff ◴[01 Apr 23 02:05 UTC] No.35396299{3}[source]▶
https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast

> This formulation of this statement has been attributed to [KentBeck][0]; it has existed as part of the [UnixWay][1] for a long time.

[0]: https://wiki.c2.com/?KentBeck

[1]: https://wiki.c2.com/?UnixWay

7. imranq ◴[01 Apr 23 02:10 UTC] No.35396337[source]▶
"Make it work, then make it right, then make it fast" is brilliant. This should be a universal principle for almost everything
8. visarga ◴[01 Apr 23 04:26 UTC] No.35397136[source]▶
> ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay

Yes. When you have to try out dozens of research ideas, most of which won't pan out, then you stop writing engineering-style code and switch to hacker mode. Why make it nice when you could be trying 2 more ideas in the meantime. Most of research code it is going to the trash anyway.

9. seydor ◴[01 Apr 23 12:26 UTC] No.35399707[source]▶
C has an almost infinite horizon for optimization. Python is good prototypes but we are beyond that stage now
replies(1): >>35400522 #
10. lostmsu ◴[01 Apr 23 14:21 UTC] No.35400522[source]▶
99% of LLM evaluation with PyTorch was already done in C++.

These .cpp projects don't improve anything for performance. They just drop dependencies necessary for training and experimentation.

replies(1): >>35400556 #
11. seydor ◴[01 Apr 23 14:29 UTC] No.35400556{3}[source]▶
Optimization isn't just about speed. As you said, dropping dependencies makes it portable, embeddable, more versatile
replies(1): >>35405181 #
12. jart ◴[01 Apr 23 23:16 UTC] No.35405181{4}[source]▶
It's also nice to not lose your mind over how crazy Python and Docker are, when all you want to do is run inference in a shell script as though it were the `cat` command. That sacred cow is going to have to come out of the temple sooner or later, and when that happens, people are going to think, wow, it's just a cow.
replies(1): >>35442417 #
13. heap_perms ◴[02 Apr 23 21:07 UTC] No.35415209{3}[source]▶
That sounds very interesting. Anyone know where I can find more on this story?
replies(1): >>35430251 #
14. buzzert ◴[03 Apr 23 19:50 UTC] No.35430251{4}[source]▶
I learned this from Steven Levy's great book, "In the Plex": https://www.amazon.com/Plex-Google-Thinks-Works-Shapes/dp/14...
15. Max-Limelihood ◴[04 Apr 23 16:43 UTC] No.35442417{5}[source]▶
Have you tried Julia for this instead?