Open Mixture-of-Experts (MoE) models share one idea but disagree on almost every routing detail. Expert counts range from 8 to 384. Top-k ranges from 1 to 8. Some models keep an always-on shared expert; others drop it.
TL;DR
- Size: 10 models, from Qwen3.6-35B-A3B (35B total, 3B active) to DeepSeek-V4-Pro (1.6T total, 49B active).
- Runs on: gpt-oss-120b fits one 80 GB GPU with MXFP4; DeepSeek-V4-Pro and Kimi K2 need multi-GPU expert parallelism.
- Performance: Routed sparsity spans 4x in Mixtral (8 experts, top-2) to 128x in Llama 4 Maverick (128, top-1).
- Best: Kimi K2 found sparsity 48 needed 1.69x fewer FLOPs than sparsity 8 at equal validation loss.
- Worst: Llama 4 Maverick and gpt-oss do not publish their gating or balancing method in config.json.
- Bottom line: Best thing: configs make routing fully auditable. Worst thing: names like “17B active” hide very different routers.
How do open MoE models route tokens?
Every model below uses token-choice routing: each token selects its own experts. They differ on 5 knobs. These are expert count, experts per token, shared experts, the scoring function and the load-balancing method. The table lists each knob per model.
MoE routing comparison table (November 2026)
| Model | Total / active params | Routed experts | Top-k | Shared experts | Gating | Load balancing |
|---|---|---|---|---|---|---|
| Mixtral 8x7B | 47B / 13B | 8 | 2 | 0 | Softmax over top-k | Aux loss, coef 0.02 (config) |
| gpt-oss-120b | 117B / 5.1B | 128 | 4 | 0 | Softmax over selected | Not disclosed |
| Qwen3-235B-A22B | 235B / 22B | 128 | 8 | 0 | Softmax, top-k normalized | Global-batch aux loss, coef 0.001 |
| Qwen3.6-35B-A3B | 35B / 3B | 256 | 8 | 1 | Not disclosed | Aux loss, coef 0.001 (config) |
| Llama 4 Maverick | 400B / 17B | 128 | 1 | 1 | Not disclosed | Not disclosed |
| DeepSeek-V3 | 671B / 37B | 256 | 8 | 1 | Sigmoid | Aux-loss-free bias (noaux_tc) |
| DeepSeek-V4-Pro | 1.6T / 49B | 384 | 6 | 1 | sqrtsoftplus | noaux_tc; hash routing in 3 layers |
| DeepSeek-V4-Flash | 284B / 13B | 256 | 6 | 1 | sqrtsoftplus | noaux_tc; hash routing in 3 layers |
| Kimi K2 | 1.04T / 32B | 384 | 8 | 1 | Sigmoid | noaux_tc + sequence aux, alpha 0.001 |
| GLM-4.5 | 355B / 32B | 160 | 8 | 1 | Sigmoid | Loss-free balance routing |
Sources: each model’s Hugging Face config.json, model card or technical report, verified November 7, 2026. Configs and cards for Qwen3.6 and Llama 4 omit gating. gpt-oss and Llama 4 omit the balancing method.