Why tokens per second is a bandwidth problem

Two different phases

Text generation has two phases with completely different bottlenecks, and conflating them is the source of most confused benchmark discussions.

Prefill processes your prompt. All tokens are available at once, so the GPU does large matrix multiplications and is limited by compute — by how many floating-point operations the chip can perform per second. Prefill is fast: hundreds or thousands of tokens per second.

Decode generates the response, one token at a time. Each token requires reading every active weight in the model from memory. There is very little arithmetic per byte read, so the GPU spends its time waiting. Decode is limited by memory bandwidth.

The estimate

Because decode reads the whole model per token, throughput is close to:

Efficiency in practice runs around 0.75 to 0.85 — you never reach the theoretical peak, and the KV cache adds reads on top. An M4 Pro has 273 GB/s. A 4-bit 8B model is roughly 4.5 GB. So:

That is close enough to real measurements to be genuinely useful for planning, and it explains the parts of the hardware that matter.

What this tells you about buying a Mac

Bandwidth, not GPU cores, sets your decode speed. Within a generation, the Max variant is roughly twice the Pro in bandwidth, and it generates roughly twice as fast on the same model. Extra GPU cores mostly help prefill — which matters if you paste large documents, and barely at all if you type short questions.

It also explains why base-chip Macs feel slower than their specs suggest, and why memory-bandwidth cuts in some generations were unpopular with this crowd: a bandwidth reduction is a proportional reduction in tokens per second.

Time to first token

TTFT is prefill time plus a fixed overhead. For a 1000-token prompt at 500 tokens/second of prefill, you wait about two seconds before the first word appears. Long documents change the feel of a model far more than decode speed does, and prompt caching — reusing the KV cache from a previous identical prefix — is the fix.

Mixture-of-experts changes the sum

An MoE model keeps all its parameters in memory but only reads a fraction per token. A 30B model with 3B active parameters needs 30B worth of memory and decodes at roughly 3B speed. That is why MoE models feel disproportionately fast on Macs with plenty of RAM, and why our estimates use active parameters for speed but total parameters for fit.

Why our numbers say "est."

Every speed figure derived this way is a model, not a measurement. Real throughput moves with the runtime version, sampler settings, batch size, prompt length, and whether your Mac is thermally throttled on battery. Where the community has submitted measured runs, we show those instead, with the sample size.