Nvidia boosts moe inference speed on dgx spark by 42.5% using three key optimizations

NVIDIA detailed a method to significantly accelerate mixture-of-experts (MoE) inference on the DGX Spark desktop system, achieving a 42.5% throughput increase when running Ant Group inclusionAI’s Ling-3.0-flash model across 1,319 GSM8K questions. This performance gain was realized by combining three distinct optimizations: a new Humming MoE kernel backend, an online FP8 layer for the language model head, and MTP speculative decoding utilizing the model's built-in NEXTN layer. While the baseline ran at 36.09 tokens per second, the combined optimizations pushed throughput to 53.89 tokens per second on the GSM8K benchmark, reducing time per output token from 26.45 ms to 18.56 ms. The optimizations work synergistically; Humming alone yielded a 9.3% boost to 39.45 tokens per second, and adding the FP8 LM head increased this to 43.94 tokens per second. The speculative decoding component provided the largest jump, increasing throughput by 185.1% to 102.88 tokens per second on synthetic data. The DGX Spark itself features a GB10 Grace Blackwell Superchip, offering 128 GB of unified memory and supporting inference for models up to 200B parameters. Ling-3.0-flash, a 124B MoE model, is available in various quantized formats, and inclusionAI lists its API at $0.075 per 1M input tokens.