←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 3 comments | 31 Mar 23 20:37 UTC | HN request time: 0.617s | source
Show context
jart ◴[31 Mar 23 21:02 UTC] No.35393615[source]▶
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
replies(24): >>35393868 #>>35393942 #>>35394089 #>>35394097 #>>35394107 #>>35394203 #>>35394208 #>>35394244 #>>35394259 #>>35394288 #>>35394408 #>>35394881 #>>35395091 #>>35395249 #>>35395858 #>>35395995 #>>35397318 #>>35397499 #>>35398037 #>>35398083 #>>35398427 #>>35402974 #>>35403334 #>>35468946 #
intelVISA ◴[31 Mar 23 22:03 UTC] No.35394288[source]▶
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :')

Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions.

Perhaps SWE is dead after all, but LLMs didn't kill it...

replies(11): >>35395112 #>>35395145 #>>35395165 #>>35395404 #>>35396298 #>>35397484 #>>35398972 #>>35399367 #>>35400001 #>>35400090 #>>35456064 #
gct ◴[01 Apr 23 02:04 UTC] No.35396298[source]▶
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
replies(1): >>35396471 #
ChatPGT ◴[01 Apr 23 02:35 UTC] No.35396471[source]▶
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately.

On the other hand, many business and professionals wouldn't exist :)

replies(1): >>35396778 #
sn_master ◴[01 Apr 23 03:23 UTC] No.35396778[source]▶
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it".

It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

replies(7): >>35397047 #>>35397059 #>>35397110 #>>35397339 #>>35397398 #>>35397582 #>>35398349 #
elbear ◴[01 Apr 23 05:23 UTC] No.35397398[source]▶
It's not the easiest syntax?

It's the easiest among most popular languages. It uses the least amount of symbols, parenthesis and braces only for values.

Some people don't like the significant whitespace, but that helps readability.

replies(2): >>35397842 #>>35398281 #
pharmakom ◴[01 Apr 23 06:51 UTC] No.35397842[source]▶
> It uses the least amount of symbols, parenthesis and braces only for values.

is there any evidence that this makes it easier?

people learn python as beginners because it has a reputation for being easy for beginners

I don't see anything about the syntax that makes it inherently easier

replies(3): >>35398511 #>>35400204 #>>35400470 #
1. whynotminot ◴[01 Apr 23 13:36 UTC] No.35400204[source]▶
What languages are you comparing it against?

Python is more readable than C. Way better than C++. Far simpler to reason about than Java. Maybe Typescript is on a similar level, but throwing a beginner into the JS ecosystem can be daunting. Perhaps Ruby could be argued as equally simple, but it feels like that's a dead end language these days. Golang is great, but probably not as easy to get rolling with as Python.

What else? Are you going to recommend some niche language no one hires for?

replies(1): >>35402694 #
2. vkou ◴[01 Apr 23 18:28 UTC] No.35402694[source]▶
> Far simpler to reason about than Java.

Strong disagreement. Explicit types make reasoning about Java much easier, especially when you are in an unfamiliar codebase.

Python is not quite the 'write-only' language of Perl, but it is a lot easier to write it than it is to read it.

replies(1): >>35403239 #
3. baq ◴[01 Apr 23 19:23 UTC] No.35403239[source]▶
Python is getting typescript like typing support. Slowly, yes, but way better than Java’s type system.