←back to thread

GGML – AI at the Edge

(ggml.ai)
899 points georgehill | 2 comments | 06 Jun 23 16:50 UTC | HN request time: 1.346s | source
Show context
yukIttEft ◴[06 Jun 23 17:39 UTC] No.36216376[source]▶
Its graph execution is still full of busyloops, e.g.:

https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9...

I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.

replies(4): >>36217006 #>>36217540 #>>36217840 #>>36218226 #
1. boywitharupee ◴[06 Jun 23 19:03 UTC] No.36217540[source]▶
is graph execution used for training only or inference also?
replies(1): >>36217851 #
2. LoganDark ◴[06 Jun 23 19:24 UTC] No.36217851[source]▶
Inference. It's a big bottleneck for RWKV.cpp, second only to the matrix multiplies.