Quantization without the hand-waving

The idea

A model is a very large pile of numbers. Trained in bf16, each of those numbers takes 2 bytes. A 7-billion-parameter model is therefore about 14 GB before anything else is loaded.

Quantization stores each number in fewer bits. At 4 bits you use one eighth of the space of bf16 — but you cannot represent much with 4 bits, so quantization works in groups. MLX takes a run of weights (64 by default), finds their minimum and maximum, and stores each weight as a small integer plus a shared scale and bias for the group.

Why 4-bit is not 0.5 bytes per weight

Those scales and biases are real data. With a group size of 64 and fp16 scale/bias, you spend 32 extra bits per 64 weights, which is 0.5 bits per weight. So nominal 4-bit costs roughly 4.5 bits in practice. Every size estimate on this site uses these effective figures:

• 3-bit: about 3.6 bits per weight
• 4-bit: about 4.5 bits per weight
• 5-bit: about 5.5 bits per weight
• 6-bit: about 6.5 bits per weight
• 8-bit: about 8.5 bits per weight
• bf16: 16 bits per weight

To convert: multiply parameters in billions by bits per weight, divide by 8, and you have gigabytes. A 7B model at 4-bit is 7 × 4.5 ÷ 8 ≈ 3.9 GB. That is why our numbers match the file sizes on Hugging Face rather than the tidy ones in blog posts.

Where quality goes

The honest summary from the community, which matches what published perplexity comparisons show:

• 8-bit is indistinguishable from the original for practical purposes. If it fits, and you care about correctness, use it.
• 6-bit is very close. A good default when you have headroom but not enough for 8.
• 4-bit is the workhorse. Most people cannot tell in chat. It shows up in long chains of reasoning, in code that must compile, and in exact recall of rare facts.
• 3-bit is a last resort. Grammar survives; reliability does not. Prefer a smaller model at 4-bit over a large one at 3-bit, almost always.

The rule that actually matters

Given a fixed memory budget, a larger model at lower precision usually beats a smaller model at higher precision — until you reach 3-bit, where the relationship inverts. A 14B at 4-bit will outperform a 7B at 8-bit at roughly the same size. A 32B at 3-bit will often not outperform a 14B at 4-bit, despite being larger.

Group size

You will see repos labelled 4bit-gs32 or similar. A smaller group means more scales, so a slightly bigger file and slightly better fidelity. gs32 at 4-bit costs about 5 bits per weight. It is a reasonable trade if you are quantizing yourself:

What quantization does not change

Speed comes from bytes moved, so a smaller model is a faster model — but quantization does not reduce the KV cache, does not extend the context window, and does not make a model smarter. It only makes it fit.