Aliyun unveils Qwen3.8-Flash open source: 125b parameter moe model with 1/9 training cost

Aliyun's Tongyi team has released the Qwen3.8-Flash model, along with the open-source weights for Qwen3.8-Flash-Next, which serves as a prototype for the upcoming Qwen4 series. The model is a multimodal MoE model with a main parameter count of 125B and an additional 51B N-gram Embedding, activating 6B parameters per token. It supports up to 262,144 tokens of context and can be extended to 1M tokens using YaRN. The attention mechanism combines GDN and QSA, achieving up to 7.6x and 4.9x acceleration in prefill and decode phases for 1M token sequences. The residual mechanism introduces Gated Residual (GR), expanding residual streams to four parallel branches with dynamic gate control. The embedding layer uses 51B parameters for N-gram Embedding, which can be offloaded to host memory for asynchronous prefetching. The optimization employs Muon Optimizer with tailored strategies for orthogonality, parameter division, and matrix splitting, and a new scaling law was fitted for the architecture. Compared to Qwen3.7-Plus, the training cost of Qwen3.8-Flash is approximately 1/9, yet it shows stronger performance in coding and office tasks. The Qwen3.8-Flash-Next-Base model achieves top results in eight out of 14 benchmarks, including MMLU-Pro, SuperGPQA, BBH, SWEBench-Pretrain, MGSM, and MMMU. The model will be available on the Qwen AI platform with API pricing at 1 yuan per million input tokens and 3 yuan per million output tokens. The model weights and FP8 quantized version were released on Hugging Face and ModelScope on August 26, 2024, at 23:00 UTC, with Qwen3.8-Flash-Next defaulting to 1M context and including official tools. The technical report and blog are available at the provided links.