←back to thread

Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon

(github.com)
268 points dipampaul17 | 1 comments | 16 May 25 20:04 UTC | HN request time: 0.375s | source

I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality.

I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising:

- K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality

This means you can run LLMs with 2-3× longer context on the same Mac. Memory usage scales with sequence length, so savings compound as context grows.

Implementation was straightforward: 1. Added --kvq-key and --kvq-val flags to llama.cpp 2. Applied existing quantization logic separately to K and V tensors 3. Validated with perplexity metrics across context lengths 4. Used Metal for acceleration (with -mlong-calls flag to avoid vectorization issues)

Benchmarked on an M4 MacBook Pro running TinyLlama with 8K context windows. Compatible with Metal/MPS and optimized for Apple Silicon.

GitHub: https://github.com/dipampaul17/KVSplit

Show context
matheist ◴[16 May 25 20:53 UTC] No.44009741[source]▶
Looks interesting! Is there any intuition for why this should be the case? Did you discover it via that intuition, or just random experimentation?

A note, your install script appears to still have a placeholder at the "apply patch" step. A suggestion, might be more user-friendly to fork llama.cpp and then include that as a git submodule rather than make it a "git clone and apply patch" step.

A further note, everyone and their dog has a different local python set-up, might be nice to let people separate the llama.cpp stuff from the python stuff rather than bake in a dependence on homebrew python.

replies(2): >>44011484 #>>44012275 #
1. Aurornis ◴[17 May 25 05:42 UTC] No.44012275[source]▶
> A note, your install script appears to still have a placeholder at the "apply patch" step. A suggestion, might be more user-friendly to fork llama.cpp and then include that as a git submodule rather than make it a "git clone and apply patch" step.

The patch doesn't actually apply to llama.cpp because argument parsing was moved to arg.cpp 8 months ago.

That doesn't matter, though, because the options to set K and V quantization were added to llama.cpp in 2023.

I don't understand why the patch exists at all, other than as an attempt to make this look novel by changing the settings through a different command line argument?

I would strongly recommend that nobody run an install.sh file from a new repo like this, especially when it's not necessary for something as simple as applying a patch file.