# opendatalab/omnidocbench architecture

> Architecture diagram of the GitHub repository opendatalab/omnidocbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendatalab/omnidocbench
- Repository: https://github.com/opendatalab/omnidocbench

## Overview

##### Project's Purpose and Type
- OmniDocBench is a benchmark and evaluation framework. Its purpose is to assess different document parsing tools and models by comparing their outputs on a richly annotated ground truth dataset.
- It supports diverse tasks (end-to-end, layout detection, recognition, etc.) and is intended for evaluation research as well as for comparing several pipelines (including expert models and vision language models).

##### File Structure to Identify Main Components
- Top-level directories such as “configs”, “dataset”, “demo_data”, “metrics”, “task”, “tools”, “utils”, and “registry” show that the project is organized into multiple concerns:
– The “configs” directory holds YAML files that configure the evaluation for different tasks (end2end, md2md, OCR, formula, table recognition, layout and formula detection).
– The “dataset” folder contains Python modules that help load and process the OmniDocBench dataset.
– “demo_data” contains sample ground truth and prediction data files for different evaluations.
– “metrics” includes modules computing various evaluation metrics (such as Edit Distance, BLEU, METEOR, TEDS, COCODet-based metrics).
– “task” encapsulates the evaluation procedures that coordinate specific tasks (detection, end2end evaluation, recognition, etc.).
– “tools” contains utility scripts for converting JSON to Markdown, visualizing outputs, and generating result leaderboard tables as well as model inference scripts.
– “utils” implements common functions for data pre-processing, matching, OCR handling, table conversion, etc.
– “registry” seems to be used for registering evaluation modules or configurations.

##### Understand Data Flow and Relationships
- At the entry point (for example, through the “pdf_validation.py” script), configuration files are loaded (from the “configs” folder) to specify dataset paths, evaluation metrics, match methods, and filtering criteria.
- The dataset information (located in “dataset”, “demo_data”, and even ground truth JSON files in OmniDocBench) is processed and fed into the evaluation modules (in “task”).
- The evaluation modules use functions from “metrics” and “utils” to calculate performance — for each subtask such as end-to-end evaluation, OCR text recognition, formula recognition, and layout detection.
- Once the evaluations are performed, results are stored in the “result” folder, and further processed for leaderboard visualization via tools like “generate_result_tables.ipynb”.
- The “tools/model_infer” subdirectory contains scripts to call various models and convert model outputs into Markdown predictions that are later compared with the ground truth.

##### Architectural Patterns and Technologies
- The design follows a modular and configuration-driven architecture. Each evaluation method (end2end, md2md, OCR, table, etc.) is isolated in its own configuration and task module.
- There are clear input and output boundaries: input dataset files and model predictions are given as JSON or Markdown files, and output evaluations (JSON files with metrics) are stored in the result directory.
- The use of mapping (in “configs” for both layout and formula detection, for example) demonstrates a pattern of transforming raw predictions to a standardized evaluation format.
- The system leverages external tools such as LaTeXML for converting LaTeX tables to HTML and has integrations for matching methods (e.g., “quick_match” or “simple_match”) to reduce segmentation differences.

##### Final Detailed Elements the Engineer Must Include
- Clear labels for each component/module
- Annotations that explain specific design decisions: for example, how “quick_match” reduces paragraph segmentation issues, or how integration with LaTeXML converts LaTeX to HTML for table evaluation.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        CFG["Config Loader (YAML)"]:::config
        DS["Dataset Loader"]:::ingestion
        DD["Demo Data"]:::ingestion
        PRE["Pre-processing Utilities"]:::ingestion
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        TM["Task Manager"]:::task
        MI["Model Inference Scripts"]:::task
        REG["Module Registry"]:::task
        E2E["End-to-End Eval"]:::task
        DET["Detection Eval"]:::task
        REC["Recognition/Formula Eval"]:::task
    end

    %% Reporting Layer
    subgraph "Reporting Layer"
        ME["Metrics Engine"]:::metrics
        CONV["Output Converter (JSON to Markdown)"]:::output
        RES["Result Aggregator (Leaderboard)"]:::output
        VIS["Visualization Tools"]:::output
    end

    %% Data Flow Connections
    CFG --> TM
    DS  --> TM
    DD  --> TM
    PRE --> TM

    MI  --> TM
    REG --> TM

    TM --> E2E
    TM --> DET
    TM --> REC

    E2E --> ME
    DET --> ME
    REC --> ME

    ME --> CONV
    ME --> RES
    ME --> VIS

    %% Click Events
    click CFG "https://github.com/opendatalab/omnidocbench/tree/main/configs/"
    click DS "https://github.com/opendatalab/omnidocbench/tree/main/dataset/"
    click DD "https://github.com/opendatalab/omnidocbench/tree/main/demo_data/"
    click PRE "https://github.com/opendatalab/omnidocbench/blob/main/utils/data_preprocess.py"
    click E2E "https://github.com/opendatalab/omnidocbench/blob/main/task/end2end_run_eval.py"
    click DET "https://github.com/opendatalab/omnidocbench/blob/main/task/detection_eval.py"
    click REC "https://github.com/opendatalab/omnidocbench/blob/main/task/recognition_eval.py"
    click ME "https://github.com/opendatalab/omnidocbench/tree/main/metrics/"
    click VIS "https://github.com/opendatalab/omnidocbench/blob/main/tools/visualization.py"
    click RES "https://github.com/opendatalab/omnidocbench/blob/main/tools/generate_result_tables.ipynb"
    click CONV "https://github.com/opendatalab/omnidocbench/blob/main/tools/json2md.py"
    click MI "https://github.com/opendatalab/omnidocbench/tree/main/tools/model_infer/"
    click REG "https://github.com/opendatalab/omnidocbench/tree/main/registry/"

    %% Styles
    classDef config fill:#E6E6FA,stroke:#9370DB,stroke-width:2px;
    classDef ingestion fill:#D1E8FF,stroke:#2B82D2,stroke-width:2px;
    classDef task fill:#FFE699,stroke:#FFB900,stroke-width:2px;
    classDef metrics fill:#C3FDB8,stroke:#66B032,stroke-width:2px;
    classDef output fill:#FFCCCB,stroke:#C70039,stroke-width:2px;
```
