←back to thread

GGML – AI at the Edge

(ggml.ai)
899 points georgehill | 1 comments | 06 Jun 23 16:50 UTC | HN request time: 0s | source
Show context
nivekney ◴[06 Jun 23 17:21 UTC] No.36216106[source]▶
On a similar thread, how does it compare to Hippoml?

Context: https://news.ycombinator.com/item?id=36168666

replies(1): >>36216469 #
brucethemoose2 ◴[06 Jun 23 17:45 UTC] No.36216469[source]▶
We don't necessarily know... Hippo is closed source for now.

Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm

But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely, but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!

replies(1): >>36217701 #
smiley1437 ◴[06 Jun 23 19:14 UTC] No.36217701{3}[source]▶
>> run LLaMA 30B at almost 3 tokens/s

Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model

replies(3): >>36217877 #>>36217992 #>>36219745 #
oceanplexian ◴[06 Jun 23 19:34 UTC] No.36217992{4}[source]▶
With a single NVIDIA 3090 and the fastest inference branch of GPTQ-for-LLAMA https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/fastest-i..., I get a healthy 10-15 tokens per second on the 30B models. IMO GGML is great (And I totally use it) but it's still not as fast as running the models on GPU for now.
replies(2): >>36219157 #>>36219874 #
1. LoganDark ◴[06 Jun 23 21:10 UTC] No.36219157{5}[source]▶
> IMO GGML is great (And I totally use it) but it's still not as fast as running the models on GPU for now.

I think it was originally designed to be easily embeddable—and most importantly, native code (i.e. not Python)—rather than competitive with GPUs.

I think it's just starting to get into GPU support now, but carefully.