Mixture-of-experts: big model, small bill at inference

The architecture in one paragraph

A dense model runs every parameter for every token. A mixture-of-experts model replaces the feed-forward block in each layer with many parallel "experts" and a small router that picks a handful of them per token. A 30B model with eight experts might activate two of them, doing roughly 3B parameters of work while holding 30B in memory.

The trade, precisely

Memory: you pay for everything. The router can send the next token to any expert, so all of them must be resident. An MoE model is exactly as large on disk and in RAM as its parameter count implies.

Speed: you pay only for what is active. Decode reads active weights, so throughput tracks active parameters, not total.

This is the single most Mac-friendly architecture available. Unified memory gives you unusual capacity, and MoE converts capacity into speed. A 64 GB Mac running a 30B-A3B model gets the knowledge of a large model at the responsiveness of a small one.

Where it disappoints

MoE models are not equivalent to dense models of the same total size. A 30B model with 3B active is generally stronger than a 3B dense model and weaker than a 30B dense model — closer to the geometric middle. If your task is hard reasoning and you have the memory for a dense 30B, the dense model usually wins on quality while losing badly on speed.

Routing also adds variance. Some prompts land on experts that handle them well and some do not, so MoE output can feel less consistent across topics than a dense model's.

Reading our numbers

On model pages for MoE entries you will see two parameter figures: total and active. The memory bar uses total. The speed estimate uses active. If a page shows a 30B model fitting in 20 GB and generating at 60 tokens per second, that is not an error — it is the whole point of the architecture.

Practical guidance

• Under 32 GB, MoE options are mostly out of reach; stick with dense models around 7–14B.
• At 48–64 GB, a 30B-class MoE at 4-bit is often the best experience available on a Mac, full stop.
• At 128 GB and above, larger MoE models become viable and the speed advantage grows, because dense models of that size decode painfully slowly.
• Always check the active parameter count before believing a speed claim. "It runs a 100B model" means very little without it.