Beyond Standard Benchmarks - Evaluating AI for Real-World Tasks

Standard AI benchmarks are the primary tool for measuring model performance, yet they have significant limitations. Many are publicly available, allowing AI systems to incorporate them during training; it remains unclear what many of these tests actually measure; and numerous benchmarks contain errors or suffer from inconsistent calibration. Despite these flaws, high-quality benchmarks like ARC-AGI do reveal an upward trend in performance that appears to reflect genuine gains in practical applications. The existing benchmark landscape leaves important gaps. Current tests focus heavily on mathematics, science, coding, and logical reasoning while providing minimal insight into abilities like creative writing, sociological analysis, or business judgment. Organizations rely on these benchmarks to select AI tools, yet this approach overlooks a crucial reality: what matters is which model best serves an organization's specific needs. An emerging practice known as vibes-based benchmarking supplements standardized tests. These are personalized evaluations designed to explore how a model understands relationships between concepts—such as asking it to draw a pelican on a bicycle or compose a challenging poem. Though subjective, these assessments reveal important patterns in how different models think and operate. OpenAI's GDPval research provides a rigorous model for real-world AI evaluation. The company recruited experts with an average of 14 years of experience to create complex, authentic tasks requiring 4 to 7 hours for human experts to complete. Multiple AI models and human professionals then tackled these same tasks, with independent evaluators assessing results without knowing their source. The findings revealed substantial performance gaps: some models excel in software development or financial advisory but struggle in fields like pharmacy or real estate. Organizations deploying AI at scale cannot rely on general public benchmarks alone. Instead, they need systematic evaluations on actual work, repeated across realistic scenarios with expert human review. This rigor is essential not merely to understand average performance but to uncover each model's specific strengths, weaknesses, and decision-making biases—critical when recommendations influence hundreds or thousands of organizational choices.