A comparative analysis of routing mechanisms in leading mixture-of-experts models

Various advanced Mixture-of-Experts (MoE) models employ token-choice routing, differing across five key parameters: expert count, experts per token, shared expert presence, scoring function, and load-balancing method. DeepSeek-V3 utilizes 256 routed experts with a sigmoid scoring function, while DeepSeek-V4 increases this to 384 experts and switches to a sqrtsoftplus function, incorporating hash routing in its initial three layers. Other models present distinct architectures. Qwen3.6 employs 256 experts, including one shared expert, while Llama 4 Maverick uses a top-1 design, routing tokens to one shared and one of 128 routed experts. gpt-oss-120b utilizes a linear router over 128 experts, omitting a shared expert. Kimi K2 mirrors the DeepseekV3ForCausalLM class, scaling to 384 routed experts. Architectural choices impact performance; for instance, increasing experts per layer raises memory load and communication traffic, whereas lowering top-k trims per-token compute. GLM-4.5 favored depth with 160 experts, contrasting with the width focus of others. These configurations, sourced from model reports and Hugging Face configs verified on November 7, 2026, highlight the diverse engineering trade-offs in MoE deployment.