Most active commenters
  • codexon(4)
  • eru(3)

←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 11 comments | 31 Mar 23 20:37 UTC | HN request time: 1.037s | source | bottom
Show context
jart ◴[31 Mar 23 21:02 UTC] No.35393615[source]▶
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
replies(24): >>35393868 #>>35393942 #>>35394089 #>>35394097 #>>35394107 #>>35394203 #>>35394208 #>>35394244 #>>35394259 #>>35394288 #>>35394408 #>>35394881 #>>35395091 #>>35395249 #>>35395858 #>>35395995 #>>35397318 #>>35397499 #>>35398037 #>>35398083 #>>35398427 #>>35402974 #>>35403334 #>>35468946 #
intelVISA ◴[31 Mar 23 22:03 UTC] No.35394288[source]▶
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :')

Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions.

Perhaps SWE is dead after all, but LLMs didn't kill it...

replies(11): >>35395112 #>>35395145 #>>35395165 #>>35395404 #>>35396298 #>>35397484 #>>35398972 #>>35399367 #>>35400001 #>>35400090 #>>35456064 #
gct ◴[01 Apr 23 02:04 UTC] No.35396298[source]▶
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
replies(1): >>35396471 #
ChatPGT ◴[01 Apr 23 02:35 UTC] No.35396471[source]▶
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately.

On the other hand, many business and professionals wouldn't exist :)

replies(1): >>35396778 #
sn_master ◴[01 Apr 23 03:23 UTC] No.35396778[source]▶
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it".

It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

replies(7): >>35397047 #>>35397059 #>>35397110 #>>35397339 #>>35397398 #>>35397582 #>>35398349 #
1. codexon ◴[01 Apr 23 04:22 UTC] No.35397110[source]▶
There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.
replies(2): >>35397342 #>>35397723 #
2. eru ◴[01 Apr 23 05:11 UTC] No.35397342[source]▶
That might be true, but it seems to generally fall under the category of 'relevant 15+ years ago', doesn't it?
replies(2): >>35397459 #>>35402983 #
3. rybar ◴[01 Apr 23 05:36 UTC] No.35397459[source]▶
How do you qualify relevancy? Your own personal bubble and bias? Adoption and usage?

Pull requests and stars on github? That might be a start.

https://madnight.github.io/githut/#/pull_requests/2022/4 https://madnight.github.io/githut/#/stars/2022/4

Though you may say but but alltheprivaterepos! Then I challenge you to back up what you mean by relevance and prove python is a category of relevant 15+ years ago.

4. ElFitz ◴[01 Apr 23 06:29 UTC] No.35397723[source]▶
> Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.

Do you have any argument to support this, aside from personal bias?

replies(1): >>35403231 #
5. codexon ◴[01 Apr 23 18:58 UTC] No.35402983[source]▶
I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it.

Even if it doesn't have the best syntax now (which I doubt), the tooling and libraries make it a better choice over any language that have an edge over python syntax.

replies(1): >>35407253 #
6. codexon ◴[01 Apr 23 19:22 UTC] No.35403231[source]▶
I can make some arguments but it all boils down to personal bias and anecdotes.

The forced use of spacing to delineate blocks means you will never see a bunch of brackets eating up screen space and the common error where someone adds another line to an if statement but doesn't add braces.

Semicolons not being conventional means less screen noise and less code golf 1 liners.

The focus on imperative vs functional means you rarely ever see something like this a(b(c(d(e(f(g))))

PHP suffers greatly from poorly named standard functions on top of all of that.

Don't get me started on Ruby metaprogramming.

These are just the things I could think of off the top of my head. I do not want to spend my afternoon on this. This is just my experience looking at code for over 20 years, you either believe it or you don't. There's no scientific studies to prove that 1 syntax feature is superior.

I highly doubt that everyone chose python just because Google did. Python was a giant step in syntax compared to the competition back then, and now even if there is a new language out there right now that has a better syntax, it's not going to be better by much, and it is not going to have the tooling, libraries, or the community.

replies(1): >>35417354 #
7. eru ◴[02 Apr 23 04:45 UTC] No.35407253{3}[source]▶
> I'm arguing against the point that it clearly did have the easiest syntax compared to the competition back then and not because Google was using it.

Maybe, not sure? My point was that both the syntax and Google using it was more relevant 15 years ago than now.

(I don't have much of an opinion on the 15+ years ago thing.)

replies(1): >>35408022 #
8. codexon ◴[02 Apr 23 07:17 UTC] No.35408022{4}[source]▶
I don't see any reason for it to be less true now.

Is python syntax worse than any brand new languages like rust or go? Absolutely not. It's still better.

Did Google stop using it? I don't think so, but I also don't think people picked it just because Google did.

replies(1): >>35417974 #
9. ElFitz ◴[03 Apr 23 01:17 UTC] No.35417354{3}[source]▶
Having not been around when Python gained in popularity, and having mostly been using Node.js and Swift, this is actually quite interesting.

Thanks!

10. eru ◴[03 Apr 23 02:52 UTC] No.35417974{5}[source]▶
Python's syntax is ok.

Btw, I wish they would take some inspiration from Haskell's syntax.

Haskell also has significant whitespace, but its defined as syntactic sugar for a more traditionally syntax with curly braces and semicolons.

Approximately no-one uses that curly-brace syntax, but it's good for two things:

- silences the naysayers

- more importantly: allows you to copy-paste code even into forms that mess up your indentation.

replies(1): >>35444338 #
11. joquarky ◴[04 Apr 23 18:55 UTC] No.35444338{6}[source]▶
In a few years, none of this is going to matter anyway since it is likely we will be able to automatically translate everything cheaply.