How opendatalab/omnidocbench works
Project's Purpose and Type
• OmniDocBench is a benchmark and evaluation framework. Its purpose is to assess different document parsing tools and models by comparing their outputs on a richly annotated ground truth dataset.
• It supports diverse tasks (end-to-end, layout detection, recognition, etc.) and is intended for evaluation research as well as for comparing several pipelines (including expert models and vision language models).
File Structure to Identify Main Components
• Top-level directories such as “configs”, “dataset”, “demo_data”, “metrics”, “task”, “tools”, “utils”, and “registry” show that the project is organized into multiple concerns:
– The “configs” directory holds YAML files that configure the evaluation for different tasks (end2end, md2md, OCR, formula, table recognition, layout and formula detection).
– The “dataset” folder contains Python modules that help load and process the OmniDocBench dataset.
– “demo_data” contains sample ground truth and prediction data files for different evaluations.
– “metrics” includes modules computing various evaluation metrics (such as Edit Distance, BLEU, METEOR, TEDS, COCODet-based metrics).
– “task” encapsulates the evaluation procedures that coordinate specific tasks (detection, end2end evaluation, recognition, etc.).
– “tools” contains utility scripts for converting JSON to Markdown, visualizing outputs, and generating result leaderboard tables as well as model inference scripts.
– “utils” implements common functions for data pre-processing, matching, OCR handling, table conversion, etc.
– “registry” seems to be used for registering evaluation modules or configurations.
Understand Data Flow and Relationships
• At the entry point (for example, through the “pdf_validation.py” script), configuration files are loaded (from the “configs” folder) to specify dataset paths, evaluation metrics, match methods, and filtering criteria.
• The dataset information (located in “dataset”, “demo_data”, and even ground truth JSON files in OmniDocBench) is processed and fed into the evaluation modules (in “task”).
• The evaluation modules use functions from “metrics” and “utils” to calculate performance — for each subtask such as end-to-end evaluation, OCR text recognition, formula recognition, and layout detection.
• Once the evaluations are performed, results are stored in the “result” folder, and further processed for leaderboard visualization via tools like “generate_result_tables.ipynb”.
• The “tools/model_infer” subdirectory contains scripts to call various models and convert model outputs into Markdown predictions that are later compared with the ground truth.
Architectural Patterns and Technologies
• The design follows a modular and configuration-driven architecture. Each evaluation method (end2end, md2md, OCR, table, etc.) is isolated in its own configuration and task module.
• There are clear input and output boundaries: input dataset files and model predictions are given as JSON or Markdown files, and output evaluations (JSON files with metrics) are stored in the result directory.
• The use of mapping (in “configs” for both layout and formula detection, for example) demonstrates a pattern of transforming raw predictions to a standardized evaluation format.
• The system leverages external tools such as LaTeXML for converting LaTeX tables to HTML and has integrations for matching methods (e.g., “quick_match” or “simple_match”) to reduce segmentation differences.
Final Detailed Elements the Engineer Must Include
• Clear labels for each component/module
• Annotations that explain specific design decisions: for example, how “quick_match” reduces paragraph segmentation issues, or how integration with LaTeXML converts LaTeX to HTML for table evaluation.