←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 2 comments | 31 Mar 23 20:37 UTC | HN request time: 0.001s | source
Show context
jart ◴[31 Mar 23 21:02 UTC] No.35393615[source]▶
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
replies(24): >>35393868 #>>35393942 #>>35394089 #>>35394097 #>>35394107 #>>35394203 #>>35394208 #>>35394244 #>>35394259 #>>35394288 #>>35394408 #>>35394881 #>>35395091 #>>35395249 #>>35395858 #>>35395995 #>>35397318 #>>35397499 #>>35398037 #>>35398083 #>>35398427 #>>35402974 #>>35403334 #>>35468946 #
intelVISA ◴[31 Mar 23 22:03 UTC] No.35394288[source]▶
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :')

Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions.

Perhaps SWE is dead after all, but LLMs didn't kill it...

replies(11): >>35395112 #>>35395145 #>>35395165 #>>35395404 #>>35396298 #>>35397484 #>>35398972 #>>35399367 #>>35400001 #>>35400090 #>>35456064 #
gct ◴[01 Apr 23 02:04 UTC] No.35396298[source]▶
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
replies(1): >>35396471 #
ChatPGT ◴[01 Apr 23 02:35 UTC] No.35396471[source]▶
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately.

On the other hand, many business and professionals wouldn't exist :)

replies(1): >>35396778 #
sn_master ◴[01 Apr 23 03:23 UTC] No.35396778[source]▶
I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it".

It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.

replies(7): >>35397047 #>>35397059 #>>35397110 #>>35397339 #>>35397398 #>>35397582 #>>35398349 #
1. laichzeit0 ◴[01 Apr 23 08:18 UTC] No.35398349[source]▶
It’s not like there’s a gun to anyone’s head forcing them to use Python. The ecosystem (library, framework, IDEs) is what draws people to use it.

If there was a superior alternative that covers the breadth of the Python ecosystem I’m pretty sure no one would have any scruples in using it. A programming language and its syntax is the least interesting or complex part when it comes to solving problems. Just rattling off some amazing libraries I've used over the last few years:

https://scikit-image.org - Image processing

https://imgaug.readthedocs.io - Image augmentation

https://scikit-learn.org/stable - ML

https://pymoo.org - Multi objective optimization

https://simpy.readthedocs.io/ - Discrete event simulation

https://lifelines.readthedocs.io - Survival analysis

https://bambinos.github.io/bambi - Bayesian modeling

https://unit8co.github.io/darts/ - Time series forecasting

https://abydos.readthedocs.io/en/latest/abydos.distance.html - Basically any string distance metric you can think of

The list just goes on and on.. oh yeah, some Deep Learning libraries too, which some people find useful.

replies(1): >>35399652 #
2. HarHarVeryFunny ◴[01 Apr 23 12:19 UTC] No.35399652[source]▶
>It’s not like there’s a gun to anyone’s head forcing them to use Python. The ecosystem (library, framework, IDEs) is what draws people to use it

Sure, but that is the gun, especially (as reflected in your examples) for machine learning. The best frameworks (PyTorch, TensorFlow, JAX) are all Python, with support for other languages being an afterthought as best.

The use of scripting languages (Python, Lua - original Torch) for ML seems to have started partly because he original users were non-developers, more from a math/stats background, and partly because an interactive REPL loop is good for a field like this that is very experimental/empirical.

Does it make sense that we're now building AGI using a scripting language? Not really, but that's where we are!