Most active commenters
  • alanfranz(6)
  • pharmakom(5)

←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 25 comments | 31 Mar 23 20:37 UTC | HN request time: 0.001s | source | bottom
Show context
jart ◴[31 Mar 23 21:02 UTC] No.35393615[source]▶
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
replies(24): >>35393868 #>>35393942 #>>35394089 #>>35394097 #>>35394107 #>>35394203 #>>35394208 #>>35394244 #>>35394259 #>>35394288 #>>35394408 #>>35394881 #>>35395091 #>>35395249 #>>35395858 #>>35395995 #>>35397318 #>>35397499 #>>35398037 #>>35398083 #>>35398427 #>>35402974 #>>35403334 #>>35468946 #
intelVISA ◴[31 Mar 23 22:03 UTC] No.35394288[source]▶
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :')

Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions.

Perhaps SWE is dead after all, but LLMs didn't kill it...

replies(11): >>35395112 #>>35395145 #>>35395165 #>>35395404 #>>35396298 #>>35397484 #>>35398972 #>>35399367 #>>35400001 #>>35400090 #>>35456064 #
gct ◴[01 Apr 23 02:04 UTC] No.35396298[source]▶
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
replies(1): >>35396471 #
ChatPGT ◴[01 Apr 23 02:35 UTC] No.35396471[source]▶
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately.

On the other hand, many business and professionals wouldn't exist :)

replies(1): >>35396778 #
sn_master ◴[01 Apr 23 03:23 UTC] No.35396778[source]▶
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it".

It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

replies(7): >>35397047 #>>35397059 #>>35397110 #>>35397339 #>>35397398 #>>35397582 #>>35398349 #
1. alanfranz ◴[01 Apr 23 06:04 UTC] No.35397582[source]▶
Python is the 2nd best language for everything.

It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow.

So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection.

I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add the problems with missing/slow type checking that slows down large scale systems development).

replies(3): >>35397835 #>>35399566 #>>35399581 #
2. pharmakom ◴[01 Apr 23 06:49 UTC] No.35397835[source]▶
> Python is the 2nd best language for everything.

Huh? Why?

You can barely deploy it to Web.

it doesn't scale perfoance wise

you can't built robust abstractions

The REPL is merely OK

You can barely ship working code without containers

the syntax is hard to manipulate programmatically

Python has inertia but it's holding us back

replies(3): >>35398468 #>>35398487 #>>35399387 #
3. l33tman ◴[01 Apr 23 08:36 UTC] No.35398468[source]▶
Well for starters, web deployment isn't "everything". Python is the de-facto go-to language for research or general prototyping, where not everyone is a programming wiz keeping track of the latest trendy new compiled language. Not everyone can compile stuff even.. :)

Having said that, I've deployed two large Django projects on the web with tons of customers and it runs and scales just fine, and it's a DREAM to maintain and develop for than for example Java.. I would go so far as to say the opposite, if you haven't used Python for web deployment you've been missing out! (you lose some efficiency I'm sure but you gain other things)

replies(1): >>35399210 #
4. renox ◴[01 Apr 23 08:39 UTC] No.35398487[source]▶
You have good points but "the syntax is hard to manipulate programmatically"??

Maybe you haven't noticed but Lisp is now a tiny niche and most new languages aren't homoiconic either..

replies(2): >>35399215 #>>35407814 #
5. pharmakom ◴[01 Apr 23 11:01 UTC] No.35399210{3}[source]▶
I was talking about running in the Web browser. it's not everything, but it's an important part of everything in my book.
replies(1): >>35402261 #
6. pharmakom ◴[01 Apr 23 11:03 UTC] No.35399215{3}[source]▶
I don't think that proves anything. If we had "JavaLisp" in the browser instead of JavaScript then Lisp would be very popular. Besides that, Python is harder to manipulate than many non-Lisps, such as JavaScript and Go.
replies(1): >>35399805 #
7. alanfranz ◴[01 Apr 23 11:31 UTC] No.35399387[source]▶
What is another language working that well in a larger number of areas?
replies(1): >>35399619 #
8. smallerfish ◴[01 Apr 23 12:03 UTC] No.35399566[source]▶
To steal from another thread, Python is the McDonald's of languages - it's ubiquitous, it doesn't take much effort, and it's really not very good.

The trope about it being the 2nd best language for everything isn't correct. It's taught in universities because it has a very short time to gratification, and the basic syntax is quite intuitive. Academics latched onto it for ML because of some excellent libraries, and it became established as a vital part of the ecosystem from there.

But it's a nightmare to support a moderate to large codebase in production, packaging continues to be a mess, and it's full of weird quirks. Great for weekend projects, but for pete's sake take a minute and port them into something more reliable before going to production with them.

replies(1): >>35399858 #
9. sigi64 ◴[01 Apr 23 12:07 UTC] No.35399581[source]▶
Python, the language with global interpret lock, Is not the 2nd best language for everything, especially in the age od multicore processors.
replies(1): >>35403218 #
10. pharmakom ◴[01 Apr 23 12:13 UTC] No.35399619{3}[source]▶
Clojure

JavaScript

Typescript

OCaml

Haskell

F#

replies(3): >>35401049 #>>35401969 #>>35433667 #
11. renox ◴[01 Apr 23 12:38 UTC] No.35399805{4}[source]▶
Python became popular without being the 'web language', the Lisps didn't.
replies(1): >>35399840 #
12. pharmakom ◴[01 Apr 23 12:42 UTC] No.35399840{5}[source]▶
Curly brace languages are more popular again.
13. alanfranz ◴[01 Apr 23 12:44 UTC] No.35399858[source]▶
I think you’re focusing too much on the letter, rather than the idea.
14. alanfranz ◴[01 Apr 23 15:31 UTC] No.35401049{4}[source]▶
Any JVM language or .NET language will take more to interface with native libraries, it’s not the same.

Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that.

JS and TS, could be. But are they so much better than Python, if better at all?

replies(2): >>35402387 #>>35416211 #
15. xxpor ◴[01 Apr 23 17:10 UTC] No.35401969{4}[source]▶
Outside of typescript, this feels like a response from a decade ago, when Python was still mired in the 2 vs 3 problem.

What's happened to the popularity of all of these languages since 2010? Outside of JS/TS, absolutely nothing. If anything, they've lost mindshare.

16. sireat ◴[01 Apr 23 17:43 UTC] No.35402261{4}[source]▶
https://github.com/pyodide/pyodide is pretty amazing for running Python client side in the browser.

You could run notebooks entirely client side https://jupyterlite.readthedocs.io/en/latest/

The startup is slow but otherwise it is pretty functional.

17. revelio ◴[01 Apr 23 17:57 UTC] No.35402387{5}[source]▶
Native library interfacing isn't really Python's strong suit, interpreter plugins are quite painful to write.

.NET has P/Invoke which is much nicer.

JVM is getting Panama+jextract, which is the nicest yet. You can go straight from header files to pure Java bindings which don't need any extra native code at all. But it's not shipped yet :(

replies(2): >>35403179 #>>35405781 #
18. baq ◴[01 Apr 23 19:18 UTC] No.35403179{6}[source]▶
Python has had cffi since figuratively forever, so I’m not sure why you compare native modules to P/Invoke?
replies(1): >>35408954 #
19. baq ◴[01 Apr 23 19:21 UTC] No.35403218[source]▶
Python is the practical language for when you do your cpu intensive tasks outside of it as a feature, since the GIL isn’t a problem with io parallelism.

You’d do better complaining about still nascent, compared to alternatives, async support or lack of jit in the official implementation.

20. alanfranz ◴[02 Apr 23 00:36 UTC] No.35405781{6}[source]▶
What is an “interpreter plugin?” Writing a Python C extension is not that painful, it’s quite well supported. And you’ve got cffi and ctypes as well.
21. kazinator ◴[02 Apr 23 06:41 UTC] No.35407814{3}[source]▶
Here, we can set Lisp aside and take grandparent comment's definition of syntax to be concrete, character-level syntax.

Python concrete syntax is harder to manipulate programmatically compared to Javascript concrete syntax.

For instance, to insert one statement into another, we need to traverse the lines of that syntax and add the right amount of indentation. We can't just plant the syntax into the desired spot and be done with it.

22. revelio ◴[02 Apr 23 10:05 UTC] No.35408954{7}[source]▶
Most important libraries with native components are using C extensions, not cffi.
23. nequo ◴[02 Apr 23 22:58 UTC] No.35416211{5}[source]▶
> Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that.

The impression about Haskell’s nicheness compared with OCaml prevails. But Haskell has a larger userbase and a larger library ecosystem than OCaml.

replies(1): >>35479230 #
24. chpatrick ◴[04 Apr 23 01:08 UTC] No.35433667{4}[source]▶
I've been using Haskell professionally for 8 years and its ecosystem is laughable compared to Python.
25. alanfranz ◴[07 Apr 23 07:47 UTC] No.35479230{6}[source]▶
A few years have passed since I last tried out both languages. Ocaml was sort of approachable, while Haskell required quite a different mindset imho, hence the “nicheness” from the general usage standpoint.