AMD Releases Instella-MoE: Fully Open Mixture-of-Experts Model Trained on Instinct GPUs

AMD has released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs. With 16 billion total parameters, the model activates only 2.8 billion per token. AMD is publishing weights from every training stage alongside data mixtures, training configurations, and inference code, distributed under a ResearchRAIL license for academic and research purposes. The model introduces two significant structural innovations. Gated Multi-head Latent Attention adds a learned output gate to multi-head latent attention, enhancing output modulation. FarSkip-Collective passes outdated and partial activations through MoE and attention layers, overlapping expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to 39.2% reduction in time-to-first-token when serving with expert parallelism. Pre-training spanned 7.1 trillion tokens from open corpora, followed by a long-context stage extending the window from 4K to 64K tokens. Post-training includes SFT, DPO, and RL in the Miles framework, achieving a base average score of 76.7 across benchmarks, surpassing other fully open models including Moonlight and SmolLM3.