Supabase Open-Sources Evals, Benchmark for Testing AI Agents

Supabase released Supabase Evals as open-source software, a framework for evaluating AI agents on practical tasks within its ecosystem. The benchmark runs models including Claude Code, Codex, and OpenCode against real scenarios ranging from database schema creation to RLS policy repair and Edge Function debugging. The project organizes tests across three dimensions: products (database, authentication, storage, edge functions, etc.), topics (security, SQL migrations, SDKs), and stages (build, deploy, investigate, resolve). Each scenario executes in realistic production-like environments, with agents making actual calls to real MCP servers and CLI tools. Initial results show that Opus 5 and Kimi K3 achieve 100% in the build stage without auxiliary skills. Sonnet 5 improved from 78% to 100% when skills were loaded, while GPT-5.6 Sol rose from 89% to 100%. The framework surfaced three recurring weaknesses: agents hand-writing migrations instead of using declarative schemas, manually verifying authentication without leveraging specialized libraries, and showing significant variation in documentation usage patterns across different models.