←back to thread

GGML – AI at the Edge

(ggml.ai)
899 points georgehill | 4 comments | 06 Jun 23 16:50 UTC | HN request time: 0s | source
Show context
world2vec ◴[06 Jun 23 17:25 UTC] No.36216161[source]▶
Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]?

[0] https://github.com/geohot/tinygrad

replies(2): >>36216187 #>>36218539 #
qeternity ◴[06 Jun 23 17:27 UTC] No.36216187[source]▶
No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp
replies(2): >>36216244 #>>36216266 #
1. stri8ed ◴[06 Jun 23 17:30 UTC] No.36216244[source]▶
How does the quantization happen? Are the weights preprocessed before loading the model?
replies(2): >>36216303 #>>36216321 #
2. sebzim4500 ◴[06 Jun 23 17:35 UTC] No.36216303[source]▶
Yes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding.

There are advantages to simplicity, after all.

replies(1): >>36216416 #
3. ggerganov ◴[06 Jun 23 17:36 UTC] No.36216321[source]▶
The weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix multiplication at integer level, carefully optimizing for both compute and memory bandwidth. Similar strategies are applied when running GPU inference - using custom kernels for fast Matrix x Vector multiplications
4. brucethemoose2 ◴[06 Jun 23 17:42 UTC] No.36216416[source]▶
Its not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684