The KV cache is the thing that surprises people
6 minute read
Your model fit at 4K context and died at 128K. This is why, and how to calculate it in advance.
Mistral 7B Instruct v0.3 on your M4 Pro with 48 GB — move the context and watch the cache grow.
Why a cache exists at all
Generating a token requires attending to every previous token. Done naively, producing token 1000 would mean recomputing the key and value projections for the 999 tokens before it — and doing that again for token 1001. The KV cache stores those projections instead, so each new token costs one forward pass rather than a thousand.
It is a pure speed-for-memory trade, and the memory side of that trade grows linearly with your conversation.
The formula
For each token, for each layer, you store a key vector and a value vector:
bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_elementTake Llama 3.1 8B: 32 layers, 8 KV heads, 128 head dimensions, fp16 cache.
2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB per tokenAt 8K context that is 1 GB. At 32K it is 4 GB. At the full 128K it is 16 GB — more than the 4-bit weights themselves. This is the entire explanation for "it worked yesterday and today it crashed": yesterday's conversation was shorter.
Grouped-query attention is the reason it is survivable
Older models used one KV head per attention head. Modern ones share: Llama 3.1 8B has 32 attention heads but only 8 KV heads, cutting the cache by a factor of four. When you compare two similar-sized models and one has a dramatically smaller cache, grouped-query attention is usually why. It is worth checking before you commit to a model for long-context work.
Levers you can pull
Ask for less context. The largest single win, and usually free. Most chat does not need 128K. Our calculator defaults to 8K for exactly this reason.
Quantize the cache. MLX can store K and V at 8 bits, halving cache memory for a small quality cost that is generally less noticeable than quantizing the weights further:
mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
--kv-bits 8 --max-kv-size 16384 --prompt "..."Cap the cache. --max-kv-size gives you a rotating window: the oldest tokens fall out rather than the process falling over. For long chats where early turns do not matter, this is the pragmatic option.
Pick a model with fewer KV heads. Structural, not tunable, but it is the difference between 128 KB and 512 KB per token.
Reading the bar
On every memory bar on this site, the KV cache is the translucent block to the right of the weights. Drag the context slider and watch it grow. If a model fits at 4K and overflows at 32K, that block is the reason, and you now know all four ways to shrink it.