←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.203s | source
Show context
abujazar ◴[31 Mar 23 22:31 UTC] No.35394638[source]▶
I love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.
replies(2): >>35395088 #>>35398259 #
faitswulff ◴[31 Mar 23 23:18 UTC] No.35395088[source]▶
How so?
replies(2): >>35395298 #>>35399707 #
kccqzy ◴[31 Mar 23 23:43 UTC] No.35395298[source]▶
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things.

This is totally the right way. Make it work, then make it right, then make it fast.

replies(5): >>35395814 #>>35395908 #>>35396260 #>>35396337 #>>35397136 #
1. visarga ◴[01 Apr 23 04:26 UTC] No.35397136[source]▶
> ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay

Yes. When you have to try out dozens of research ideas, most of which won't pan out, then you stop writing engineering-style code and switch to hacker mode. Why make it nice when you could be trying 2 more ideas in the meantime. Most of research code it is going to the trash anyway.