Artificial analysis and liquid AI launch mobile AI benchmark, highlighting performance of 8gb models on iPhone 17 pro

On August 24, AI evaluation organization Artificial Analysis published a blog post detailing a collaboration with Liquid AI to introduce a local AI performance benchmark for mobile devices. The benchmark focuses on the real-world performance of models on the Apple iPhone 17 Pro. The partnership is structured with distinct responsibilities: Artificial Analysis handles intelligence level testing, while Liquid AI oversees reasoning performance assessments. Five benchmark categories were selected, including function calls, instruction following, knowledge understanding, difficult questions, and mathematics. The benchmark targets models with a size of no more than 8GB after 4-bit quantization, enabling them to operate on the edge. Examples of such models include Gemma 4 E2B and LFM2-2.6B-Exp. When the context limit is set at 16K tokens, the highest average scores were achieved by Nanbeige4.2-3B from Nanhua Lab and LFM2.5-2.6B from Liquid AI. Under a one-minute response time constraint, LFM2.5-8B-A1B leads, followed by LFM2-2.6B-Exp, Gemma 4 E4B, and Granite 4.1 8B. Nanbeige4.2-3B, developed by BOSS Zhipin's laboratory, has only 3 billion non-embedded parameters but employs the Looped Transformer architecture. This architecture reuses layers cyclically, increasing effective computational depth without adding parameters, resulting in scores comparable to its competitors. The benchmark highlights the intelligence ceiling of 'edge small models,' indicating a shift in mobile AI competition from mere functionality to speed and accuracy.