←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.209s | source
1. singularity2001 ◴[31 Mar 23 22:21 UTC] No.35394515[source]▶
Does that only happen with the quantized model or also with the float16 / float32 model? Is there any reason to use float models at all?