Datalab launches OmniParseBench: a comprehensive ocr benchmark with over 16,000 tests

Datalab has launched OmniParseBench, an open-source OCR benchmark designed to evaluate document parsers. The benchmark includes 16,288 pass/fail tests across 2,937 pages from 2,343 documents, covering 92 languages. The dataset and code are publicly available, enabling engineers to assess how well parsers handle specific document types. Datalab's own parser, Datalab Accurate, ranks second in the initial leaderboard, behind Gemini 3.8 Flash. OmniParseBench evaluates parsers based on three categories: text, tables, and layout. Each test is a yes/no question about a single page, with scores calculated as the percentage of tests passed. The benchmark aims to provide a clearer, more actionable assessment than traditional leaderboards. It also includes detailed metadata, such as tags for structure, rendering, role, script, and language, allowing for customizable scoring. Datalab highlights limitations in existing benchmarks, such as the lack of standardization in scoring and overfitting to specific test conditions. The benchmark also records derivation methods, including publisher XML, PDF text layers, and human review. Datalab emphasizes that the best benchmark is one tailored to a user's own documents, encouraging vendors to use their own data for evaluation.