←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
abujazar ◴[31 Mar 23 22:31 UTC] No.35394638[source]▶
I love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.
replies(2): >>35395088 #>>35398259 #
faitswulff ◴[31 Mar 23 23:18 UTC] No.35395088[source]▶
How so?
replies(2): >>35395298 #>>35399707 #
kccqzy ◴[31 Mar 23 23:43 UTC] No.35395298[source]▶
There are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things.

This is totally the right way. Make it work, then make it right, then make it fast.

replies(5): >>35395814 #>>35395908 #>>35396260 #>>35396337 #>>35397136 #
Hortinstein ◴[01 Apr 23 00:49 UTC] No.35395814[source]▶
Not sure if you came up with "Make it work, then make it right, then make it fast." but I just screenshotted it and made it the mantra for my current project...which is by far more complicated than anything I have done as a side project. I am struggling with the desire to go "make it right" as I work on shipping the deployable prototype (right now running on cloud services I control)...thanks for this
replies(1): >>35396299 #
1. faitswulff ◴[01 Apr 23 02:05 UTC] No.35396299{3}[source]▶
https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast

> This formulation of this statement has been attributed to [KentBeck][0]; it has existed as part of the [UnixWay][1] for a long time.

[0]: https://wiki.c2.com/?KentBeck

[1]: https://wiki.c2.com/?UnixWay