←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.208s | source
Show context
detrites ◴[31 Mar 23 20:58 UTC] No.35393558[source]▶
The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?

Ok I answered my own question.

replies(5): >>35393627 #>>35393885 #>>35393921 #>>35394786 #>>35397029 #
politician ◴[31 Mar 23 21:25 UTC] No.35393885[source]▶
Roughly: OpenAIs don’t employ enough jarts.

In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas.

When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.

replies(5): >>35393957 #>>35394249 #>>35394957 #>>35396490 #>>35397974 #
dsjoerg ◴[31 Mar 23 23:04 UTC] No.35394957[source]▶
~Whats a jart?~

Ah I see https://news.ycombinator.com/user?id=jart

replies(1): >>35395087 #
sp332 ◴[31 Mar 23 23:18 UTC] No.35395087[source]▶
In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America

https://en.m.wikipedia.org/wiki/Justine_Tunney

replies(1): >>35396194 #
hackernewds ◴[01 Apr 23 01:44 UTC] No.35396194[source]▶
what's your point? also interestingly JART is in the thread here, so they might have read your comment :)
replies(1): >>35401864 #
1. sp332 ◴[01 Apr 23 16:59 UTC] No.35401864[source]▶
I'm pretty sure she knows already?