Nvidia demonstrates that harnesses, not AI models, are key to long-horizon tasks

Nvidia has released new research highlighting that the harness, rather than the AI model itself, plays a critical role in enabling AI systems to perform complex, long-horizon tasks. The harness, described as the software framework that wraps around an AI model, includes tools, memory management, and supervisory components that allow the model to function as an autonomous agent. In tests, researchers used a custom harness with a 'supervisor' component to achieve a 100% score on the ARC-AGI-3 benchmark, a set of 2D games requiring problem-solving without explicit instructions. Without the harness, the same model scored only 30%, underscoring the importance of the framework in enabling effective task execution. Adel El Hallack, vice president of product at Nvidia’s AI unit, explained that the harness is essential for turning a model into an agent, managing memory, context, and feedback. He emphasized that the harness is not just an API but a comprehensive system that includes tools, runtime, and libraries. Long-horizon tasks, which require sustained decision-making over extended periods, remain a major challenge in agentic AI research. Nvidia’s findings align with similar research from OpenAI, which also found that tweaking harness settings significantly improved performance on the same benchmark. The research also highlights broader implications for AI development, suggesting that model choice is not the sole determinant of agentic performance. Databricks previously demonstrated that different harnesses can lead to significant variations in AI costs, with the wrong harness potentially doubling expenses. Nvidia argues that open harnesses, like open models, give users greater control and flexibility, enabling more precise tuning of AI systems. El Hallack noted that open agent stacks, encompassing the harness, infrastructure, and runtime, are essential for advancing the field securely and efficiently. In addition to the main findings, the article also mentions several other developments in the AI space, including Microsoft’s research on long-horizon tasks, where models produced error-filled documents, and the shutdown of Relay, an AI automation startup. These examples illustrate the ongoing challenges and evolving landscape of agentic AI research and deployment.