Liquid AI introduces open-source benchmarking tool to evaluate on-device AI performance

Liquid AI, in collaboration with Artificial Analysis, has launched Pipette, an open-source platform designed to benchmark foundation models on edge devices. The tool measures the performance of models in real-world conditions, taking into account factors such as quantization, runtime, and hardware. Unlike traditional model cards that assess performance under server-class conditions, Pipette evaluates how models behave on devices like smartphones and tablets. The platform provides a comprehensive dataset covering over 1,000 configurations, including more than 30 models, various quantization formats, and context lengths ranging from 256 to 8,192 tokens. Initial results from devices such as a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra show that two 350M models retain 78.4% and 33.8% of decode throughput at 4,096 tokens. Pipette is released under the Apache 2.0 license and includes infrastructure components, a public dataset, a hosted dashboard, and native benchmarking apps for iOS and Android. The methodology has been validated by Artificial Analysis, ensuring the results are reliable and reproducible. The tool measures performance metrics such as latency, throughput, and memory usage, while quality assessments are tracked separately on IFBench, GPQA Diamond, and MATH-500. These quality scores are derived from evaluations on NVIDIA H100 80GB systems and matched to on-device runs with the same model and quantization. Performance tests follow a standardized methodology, including fixed token shapes, greedy decoding, and readiness gating. Each benchmark run is verified for thermal and load conditions, with only successful runs being published. The platform also supports deterministic, model-blind scoring to ensure fairness. The release includes technical details and a leaderboard, with community-submitted results currently in beta. Liquid AI encourages users to follow their social media channels and join their Telegram group for further updates.