Quantizing a model yourself

5 minute read

When the quant you need does not exist, converting it is one command and about ten minutes.

When it is worth doing

The community publishes 4-bit and 8-bit versions of popular models quickly. It is worth converting yourself when:

You need an intermediate precision — 6-bit is often the perfect fit and rarely published.
The model is new, obscure, or fine-tuned by you.
You want a specific group size for a better size-quality trade.
You want to verify the quantization rather than trust an anonymous upload.

The command

mlx_lm.convert --hf-path Qwen/Qwen3-8B \
  --mlx-path ./qwen3-8b-6bit \
  -q --q-bits 6 --q-group-size 64

This downloads the original weights, quantizes them and writes an MLX-format directory you can point mlx_lm.generate at.

What you need free

Conversion loads the original model, so you need enough memory for the full-precision weights — roughly 2 GB per billion parameters for a bf16 model. An 8B model wants about 16 GB free during conversion, even though the output is 5 GB. This is the step that catches people out on smaller Macs. You only need it once.

Choosing bits and group size

Bits dominate. Group size is a smaller lever: 32 gives slightly better fidelity and a slightly larger file than the default 64; 128 goes the other way. At 4 bits, group size 32 costs about half a bit per weight more and is usually worth it. At 8 bits it makes almost no difference.

Check the result

mlx_lm.generate --model ./qwen3-8b-6bit --prompt "Write a haiku about memory bandwidth." --temp 0.0

Run your own five test prompts at temperature 0 against both the original and the quantized version. Perplexity numbers are a useful sanity check but they do not tell you whether the model still does your job.

Share it

mlx_lm.convert --hf-path <model> -q --upload-repo <your-username>/<model>-6bit

Publishing a quant that did not exist is one of the cheapest genuinely useful contributions to this ecosystem. Include the bit width and group size in the repo name so the next person can find it.

A note on fine-tunes

If you have trained a LoRA adapter, fuse it before quantizing so the merged weights get quantized together:

mlx_lm.fuse --model <base> --adapter-path ./adapters --save-path ./fused
mlx_lm.convert --hf-path ./fused --mlx-path ./fused-4bit -q