Google's gemini 4 carbon ranks second in new multimodal knowledge agent benchmark, showing superior cost efficiency

Google, in collaboration with Artificial Analysis, has launched the Multimodal Knowledge Agent (MKA) benchmark to assess AI agents' capabilities in handling complex visual documents and real-world knowledge tasks. The new test set comprises 200 tasks, developed with domain experts across 10 industries and 83 professions, covering fields like engineering, finance, and medicine. These simulated professional projects typically require experienced personnel 3 to 8 hours to complete. Google reported that its Gemini 4 Carbon achieved the second-place ranking among participating models. Crucially, the average task cost for Gemini 4 Carbon was only one-sixth that of the top-ranked model. Rohan Doshi, Senior Product Manager at Google DeepMind, and Ben Caine, Chief Engineer, noted that many existing knowledge benchmarks rely solely on text, whereas real-world work demands visual data analysis. The evaluation framework mandates that all models run on the open-source Stirrup framework, using identical tools and sandbox environments. The final scoring is performed by an agent system built outside of Google, which verifies results against expert standards, achieving a 96% consistency rate with human scoring. While the 200 official tasks are private to prevent data leakage, 10 tasks have been released on Hugging Face for developer assessment. Gemini 4 Carbon is currently available via the Gemini API in Google AI Studio.