Context windows: advertised, usable, and affordable
5 minute read
A model that claims 128K rarely gives you 128K useful tokens, and you probably cannot afford them anyway.
Mistral 7B Instruct v0.3 on your M4 Pro with 48 GB — move the context and watch the cache grow.
Three different numbers
Advertised context is what the model card says: 128K, 256K, a million. It is the maximum the position encoding supports.
Usable context is how far in you can go before quality collapses. Independent needle-in-a-haystack testing consistently shows degradation well before the advertised limit — attention spreads thin, and material in the middle of a long input gets recalled least reliably.
Affordable context is what your KV cache budget allows. This is usually the binding constraint on a Mac, and it is the one we can calculate exactly.
Do the affordability sum first
Take the per-token cache cost from the model page, multiply by the context you want, and see whether it still fits alongside the weights. For many 8B-class models at fp16 you are spending roughly 1 GB per 8K tokens. A 32 GB Mac running a 4-bit 8B has room for a lot of context. A 16 GB Mac running the same model at 64K does not.
When you genuinely need long context
- Reading a long document you cannot chunk without losing cross-references.
- Codebase-wide questions where the answer depends on several files at once.
- Long agent loops where tool output accumulates.
When you do not
- Chat. Almost all chat fits comfortably in 8K.
- Retrieval. If you can find the relevant 2,000 tokens and pass only those, you get better answers than dumping 100,000 tokens and hoping. Retrieval beats brute force on quality as well as memory.
- Summarizing many documents. Map-reduce over chunks is cheaper and often more accurate.
Making long context cheaper
Quantize the cache to 8 bits and you halve the cost. Cap it with a rotating window and the oldest tokens drop out instead of the process crashing. Pick a model with grouped-query attention and few KV heads and the whole problem shrinks by a factor of four before you tune anything.
mlx_lm.generate --model <repo> --kv-bits 8 --max-kv-size 32768 --prompt "..."The recommendation
Set the context slider on this site to the largest context you actually use, not the one printed on the model card. Then choose among the models that fit at that setting. Choosing a model on its advertised window and discovering the cost later is the most common way people end up disappointed with a local setup.