Llama.cpp 30B runs with only 6GB of RAM now

Is there a reason Llama is getting so much attention compared to say T5 11B?

Not sure how neutral or what benchmarks are used on the following link, but T5 seems to sit a lot higher on this leaderboard?

https://accubits.com/large-language-models-leaderboard/

llama can run on an m1. T5 still needs a specialized gpu

Is llama open source? I heard it was pirated from Facebook

What is the reason T5 needs a specialized GPU and Llama doesn't?

In the end they are mathematical models, so what would prevent someone from loading T5 into a machine with plenty of RAM (like a server)? Would the codebase truly require that much refactoring? How difficult would it be to rewrite the model arhitecture as a set of mathematical equations (Einstein summation) and reimplement inference for CPU?

I did not claim llama was open source, but I see the url I posted insinuates that (probably for a contorted meaning of open source, as in source available for approved academics).

Anyway, T5 being available for download from Huggingface only makes my question more pertinent...

I made an app for running t5 locally - compiled version allows you to run without installing anything.

https://capsizegames.itch.io/chat-ai

https://github.com/Capsize-Games/chatai

I'm far from an expert in this area. But llama has been updated so anyone can hack with it on their m1 macbook (which many developers have). If someone updated T5 to be as easy to dev against, then I am sure they would see similar community interest.

Most people don't have the hardware or budget to access these specialized high vram GPUs.

interesting, what are the hardware requirements?

does it happen to run on CPU on a server with 96GB RAM?

the compiled app is meant for people to install and use with their GPU and runs on as low as a GTX 1080. I haven't tested against CPU only builds.

You can take a look at the source code and see if it would be useful to you.