Ant group releases Ling-3.0 tiny and flash base models with full training checkpoints

Ant Group's Bai Ling has officially released the Ling-3.0-tiny and Ling-3.0-flash base models, along with their pre-training and mid-training checkpoints. The release includes six checkpoints, covering pre-training, mid-training, and WSM (Warmup-Stable and Merge) stages. These checkpoints are designed for continuous pre-training, domain-specific fine-tuning, preference optimization, reinforcement learning, distillation, and research into long-context and MoE systems. The base models are not yet aligned with instruction-following tasks and are not recommended for direct deployment as chat services or in safety-critical applications without further training and evaluation. The Ling-3.0-tiny-base model has 7.9 billion total parameters and 1.3 billion activation parameters, achieving better results than its predecessor despite a 50% reduction in total parameters. The Ling-3.0-flash-base model, with 124 billion total parameters and 5.1 billion activation parameters, offers greater capacity and sparse activation design, making it suitable for real-world applications. Both models demonstrate strong performance in coding, complex reasoning, and long-context tasks, even when compared to models with twice or thrice their parameter count. The models are available on Hugging Face and ModelScope, with links provided for each checkpoint version.