Prime intellect unveils prime mega moe: a trainable fused moe kernel for enhanced training efficiency

Prime Intellect has introduced Prime Mega MoE, a trainable fused MoE kernel designed to improve training efficiency for large-scale models. This kernel builds on DeepSeek’s Mega MoE, which was integrated into its open-source DeepGEMM library in April 2026. Prime Intellect extended the kernel to support backward passes, making it suitable for full training rather than just inference. The new kernel enables up to 34% faster training steps on Qwen3 MoE models, with gains reaching 55% compared to TorchTitan. The kernel supports BF16, MXFP8 (Blackwell only), and blockwise FP8 (Hopper only), with a unique fp8_bwd mode combining BF16 forward and FP8 backward. Training results, based on 8×H200 nodes, FP8, and 16,384-token sequences, show significant improvements in throughput for both 30B and 235B models. Prime Mega MoE is now available as a dispatch backend in the prime-rl framework, with prebuilt DeepGEMM wheels provided through prime-kernels.