A comprehensive look at mixture-of-experts routing techniques in modern AI models

The selection mechanism, or router, is central to Mixture-of-Experts (MoE) models, determining which specialized expert networks process each token. Various strategies shape the quality, cost, and load balance of these systems. Techniques range from standard Top-K selection, where models like Mixtral 8x7B activate 13B parameters out of 47B, to Switch Transformer, which routes each token to a single expert while preserving quality. Other advanced methods include DeepSeekMoE, which separates common knowledge into shared experts alongside sparsely routed ones, a pattern followed by Kimi K2 and GLM-4.5. Sigmoid gating, utilized by DeepSeek-V3 and GLM-4.5, introduces per-expert bias that adjusts based on load. Furthermore, DeepSeek-V3 caps token routing to 4 nodes to optimize computation overlap, while Google’s expert choice routing in research settings allows experts to select tokens, potentially speeding up convergence by over 2x. Beyond these, hash routing assigns experts based on token ID without learning, and dynamic top-k allows variable expert activation per token. For deployment, configuration files specify parameters like expert count and topk_method, with sigmoid scoring and bias balancing currently noted as the open default for new pretraining runs.