# mbzuai-oryx/kitab-bench architecture

> Architecture diagram of the GitHub repository mbzuai-oryx/kitab-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mbzuai-oryx/kitab-bench
- Repository: https://github.com/mbzuai-oryx/kitab-bench

## Overview

##### Project Type & Purpose
- Project type: A Python-based, multi-module **benchmarking toolkit** for Arabic OCR and document-understanding tasks.

##### File Structure & README
- Root: README, LICENSE, static/ (documentation assets).
- Nine domain folders (e.g., ocr-eval, layout-eval, tables-eval, etc.). Each contains:
– eval.py / main.py: orchestrator script
– metrics.py (or metrics_utils.py): computes task-specific metrics
– models/: wrappers for each model (Tesseract, EasyOCR, Gemini, GPT-4o, Qwen, etc.)
– prompts.py (for LLM-based tasks)
– requirements.txt
– domain-specific data (e.g., pdfs-eval has pdfs/ + labels/)
- Key takeaway: **Modular, repeatable pipeline** per domain.

##### Main Components & Their Roles
A. CLI / Entry Point
– Single-command launch of each domain’s pipeline via Python scripts.
B. Data Loader
– Reads raw inputs (images, PDFs, markdown labels).
C. Model Interface Layer
– Abstracts multiple backends:
- Traditional OCR engines (Tesseract, EasyOCR)
- Vision-language and LLM APIs (GPT-4o, Gemini, Qwen-VL)
- Specialized models (RT-DETR, YOLO, Surya for layout, DocLayNet, Img2Table)
– Handles authentication (OPENAI_API_KEY, GEMINI_API_KEY)
D. Inference Engine
– Calls the chosen model wrapper, passing data & prompts, returns raw outputs.
E. Metrics Calculator
– Domain-specific metrics modules compute CER, WER, MARS, TEDS, SCRM, CODM, mAP, BLEU, etc.
F. Results Aggregator & Export
– Writes evaluation results (JSON, CSV, console logs).
G. Visualization / Reporting (optional)
– Static images (in docs) illustrate performance, but not part of runtime pipeline.

##### Interactions & Data Flow
1. User invokes `python eval.py --model_name X` in a domain folder.
2. **CLI** parses arguments → calls **Data Loader**.
3. **Data Loader** fetches input samples.
4. For each sample, pass to **Model Interface → Inference Engine**.
5. Collect predictions → feed into **Metrics Calculator**.
6. **Metrics Calculator** outputs scores → **Results Aggregator** logs them.
7. (Optional) Combine all domain results for summary performance table.

##### Architectural Patterns & Principles
- Modular Monorepo: each domain is a self-contained module.
- Strategy Pattern: Model wrappers implement a common interface (`predict()`), enabling swap-in of any engine.
- Pipeline Pattern: Clear sequence—Load → Infer → Evaluate → Report.
- Dependency Injection: Models and metrics passed as parameters to eval scripts.
- Separation of Concerns:
– Data loading, inference, and metrics are separated into distinct files/modules.
– Domain data (pdfs, labels) lives alongside its pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top-level Orchestrator
    CLI["CLI / Orchestrator\n(python eval.py)"]:::cli

    %% Domain Modules
    subgraph "Domain Modules" 
        direction TB
        OCR["ocr-eval/eval.py"]:::domain
        LAYOUT["layout-eval/main.py"]:::domain
        LINES["lines-eval/eval.py"]:::domain
        TABLES["tables-eval/eval.py"]:::domain
        CHARTS["charts-eval/eval.py"]:::domain
        DIAGRAMS["diagrams-eval/eval_diagrams.py"]:::domain
        PDFS["pdfs-eval/eval.py"]:::domain
        VQA["vqa-eval/eval.py"]:::domain
    end

    CLI -->|"invoke domain pipeline"| OCR
    CLI --> LAYOUT
    CLI --> LINES
    CLI --> TABLES
    CLI --> CHARTS
    CLI --> DIAGRAMS
    CLI --> PDFS
    CLI --> VQA

    %% Data Loader
    DataLoader["Data Loader\n(read inputs)"]:::data

    OCR --> DataLoader
    LAYOUT --> DataLoader
    LINES --> DataLoader
    TABLES --> DataLoader
    CHARTS --> DataLoader
    DIAGRAMS --> DataLoader
    PDFS --> DataLoader
    VQA --> DataLoader

    %% Inference Engine & Model Interface
    subgraph "Inference Engine\n& Model Interface" 
        direction TB
        Models["Model Wrappers"]:::model
        Prompts["Shared Prompts"]:::model
    end

    DataLoader --> Models
    DataLoader --> Prompts
    Models -->|"predict() call"| DataLoader

    %% External APIs
    subgraph "External Services"
        direction TB
        OpenAI["OpenAI API"]:::external
        Gemini["Gemini API"]:::external
    end

    Models -->|uses| OpenAI
    Models -->|uses| Gemini

    %% Metrics Calculator
    Metrics["Metrics Calculator\n(domain-specific modules)"]:::metrics
    Models --> Metrics
    DataLoader --> Metrics

    %% Results Aggregator & Dashboard
    Aggregator["Results Aggregator\n(JSON/CSV logs)"]:::aggregator
    Dashboard["Results Dashboard"]:::aggregator

    Metrics --> Aggregator
    Aggregator --> Dashboard

    %% Click Events for eval scripts
    click OCR "https://github.com/mbzuai-oryx/kitab-bench/blob/main/ocr-eval/eval.py"
    click LAYOUT "https://github.com/mbzuai-oryx/kitab-bench/blob/main/layout-eval/main.py"
    click LINES "https://github.com/mbzuai-oryx/kitab-bench/blob/main/lines-eval/eval.py"
    click TABLES "https://github.com/mbzuai-oryx/kitab-bench/blob/main/tables-eval/eval.py"
    click CHARTS "https://github.com/mbzuai-oryx/kitab-bench/blob/main/charts-eval/eval.py"
    click DIAGRAMS "https://github.com/mbzuai-oryx/kitab-bench/blob/main/diagrams-eval/eval_diagrams.py"
    click PDFS "https://github.com/mbzuai-oryx/kitab-bench/blob/main/pdfs-eval/eval.py"
    click VQA "https://github.com/mbzuai-oryx/kitab-bench/blob/main/vqa-eval/eval.py"

    %% Click Events for Prompts & Aggregator
    click Prompts "https://github.com/mbzuai-oryx/kitab-bench/blob/main/charts-eval/prompts.py"
    click Prompts "https://github.com/mbzuai-oryx/kitab-bench/blob/main/pdfs-eval/prompts.py"
    click Prompts "https://github.com/mbzuai-oryx/kitab-bench/blob/main/tables-eval/prompts.py"
    click Prompts "https://github.com/mbzuai-oryx/kitab-bench/blob/main/vqa-eval/prompts.py"
    click Aggregator "https://github.com/mbzuai-oryx/kitab-bench/blob/main/diagrams-eval/get_results_diagrams.py"

    %% Styles
    classDef cli fill:#D5DBDB,stroke:#333,stroke-width:1px
    classDef domain fill:#D6EFFF,stroke:#333,stroke-width:1px
    classDef data fill:#FADBD8,stroke:#333,stroke-width:1px
    classDef model fill:#FFE5B4,stroke:#333,stroke-width:1px
    classDef metrics fill:#D5F5E3,stroke:#333,stroke-width:1px
    classDef external fill:#E8DAEF,stroke:#333,stroke-width:1px
    classDef aggregator fill:#FDEBD0,stroke:#333,stroke-width:1px
```
