Keenable AI launches open-source benchmark for real-time web search evaluation

Keenable AI has introduced NEEDLE, an open-source benchmark designed to evaluate web search APIs by continuously updating its query set from fresh public sources. Unlike traditional benchmarks that use static datasets, NEEDLE regenerates queries hourly for news and daily for finance, scholar, legal, and rare-entity topics. The benchmark tests fifteen search APIs under a unified protocol, with results compared against an ultimate oracle engine that aggregates all retrieved information. NEEDLE, a Python CLI tool, requires an OpenRouter key and API keys for each tested engine, and runs on laptops or CI systems. It evaluates both query streams and ranking quality without fetching pages or re-ranking results. The benchmark’s scoring system varies by domain, with news and deep-tail queries rated by LLM judges, while finance and legal tasks use identifier matches. Results from the past seven days show Exa leading in finance with 0.910, while Keenable and Perplexity closely follow. In scholar and legal domains, Keenable scores 0.774 and 0.470 respectively, against a ceiling of 0.869. Latency metrics also highlight performance differences, with Keenable-realtime at 193 ms p50. The project’s live dashboard, GitHub repo, and technical write-up are publicly available.