←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 3 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source
Show context
DoctorOetker ◴[01 Apr 23 14:41 UTC] No.35400653[source]▶
Is there a reason Llama is getting so much attention compared to say T5 11B?

Not sure how neutral or what benchmarks are used on the following link, but T5 seems to sit a lot higher on this leaderboard?

https://accubits.com/large-language-models-leaderboard/

replies(2): >>35400879 #>>35400909 #
barbariangrunge ◴[01 Apr 23 15:14 UTC] No.35400909[source]▶
Is llama open source? I heard it was pirated from Facebook
replies(1): >>35401300 #
DoctorOetker ◴[01 Apr 23 15:59 UTC] No.35401300[source]▶
I did not claim llama was open source, but I see the url I posted insinuates that (probably for a contorted meaning of open source, as in source available for approved academics).

Anyway, T5 being available for download from Huggingface only makes my question more pertinent...

replies(1): >>35403614 #
1. w4ffl35 ◴[01 Apr 23 20:08 UTC] No.35403614[source]▶
I made an app for running t5 locally - compiled version allows you to run without installing anything.

https://capsizegames.itch.io/chat-ai

https://github.com/Capsize-Games/chatai

replies(1): >>35405400 #
2. DoctorOetker ◴[01 Apr 23 23:41 UTC] No.35405400[source]▶
interesting, what are the hardware requirements?

does it happen to run on CPU on a server with 96GB RAM?

replies(1): >>35411916 #
3. w4ffl35 ◴[02 Apr 23 15:58 UTC] No.35411916[source]▶
the compiled app is meant for people to install and use with their GPU and runs on as low as a GTX 1080. I haven't tested against CPU only builds.

You can take a look at the source code and see if it would be useful to you.