# microsoft/table-transformer architecture

> Architecture diagram of the GitHub repository microsoft/table-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/table-transformer
- Repository: https://github.com/microsoft/table-transformer

## Overview

##### Main Components
- Data Sources & Ingestion:
- External Data: PubTables-1M dataset, FinTabNet, SciTSR, etc. (downloaded from Microsoft Research Open Data or Hugging Face).
- Pre-processing Scripts: Located in the "scripts" directory (e.g., process_fintabnet.py, process_icdar2013.py) that prepare raw data.

- Data Pipeline & Configuration:
- Data configuration files in "src" (detection_config.json and structure_config.json) that define hyperparameters and operational settings.
- The environment management via environment.yml and conda setup.

- Model Components and Training:
- Training codes: “main.py” in both the “detr” and “src” directories (with flags to choose between “detection” and “structure” tasks) indicate the training pipeline.
- Supporting utilities: “detr/util” contains helper routines (e.g., box_ops, plot_utils).

- Inference and Evaluation:
- Inference Scripts: “src/inference.py” provides a simple pipeline for applying the trained model to input images.
- Evaluation Modules: “src/eval.py” and “test_all.py” in the detr folder which compute detection metrics (AP, AR) as well as GriTS metrics for table structure recognition.
- Debug/Visualization: The debug functionality in the training scripts (flags for debug visualizations) shows how inference outputs and intermediate results are logged.

- External Integrations/Dependencies:
- Pre-trained weights hosted on Hugging Face (links provided in the README).
- External libraries: PyTorch, Torchvision, and likely detectron2 (in the detr/d2 sub-component).

##### Relationships and Data Flow
- Data flows from external sources to pre-processing scripts, then becomes input for both training and evaluation modules.
- Training and inference modules depend on configuration files, which govern hyperparameters and dataset paths.
- Utilities and debug outputs support the training and evaluation loops.
- External sources (like Hugging Face for weights or Microsoft Research Open Data for the raw dataset) can be shown as separate blocks feeding into the system.

##### Architectural Patterns & Design Principles
- Modularity: Separate directories for models, datasets, engine (training/evaluation loops), and utilities keep the design modular.
- Config-driven: The use of JSON config files and environment setup shows dependency injection which allows different training scenarios.
- Pipeline/Flow Architecture: The separation between data ingestion, training, and inference indicates a pipeline-oriented design.
- Reuse of components: The architecture is designed to be extensible (e.g., different model weights, possibility of fine-tuning by loading checkpoints).

##### Relevant Technologies to Highlight
- Deep Learning Framework: PyTorch/Torchvision is at the core of training and inference.
- Docker & CI/CD: Presence of a Dockerfile (for environment consistency) and GitHub workflows (CI) in the “.github” directory.
- Configuration Management: JSON config files and environment.yml for reproducibility.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data Sources
    subgraph "External Data Sources"
        P1["PubTables-1M"]:::external
        F1["FinTabNet"]:::external
        S1["SciTSR"]:::external
        MS1["Microsoft Research Open Data"]:::external
        HF1["Hugging Face"]:::external
    end

    %% Data Pre-processing Scripts
    subgraph "Data Pre-processing"
        PF1["process_fintabnet.py"]:::process
        PI1["process_icdar2013.py"]:::process
        PP1["process_pubmed.py"]:::process
        PS1["process_scitsr.py"]:::process
        VA1["view_annotations.py"]:::process
    end

    %% Data Pipeline & Configuration
    subgraph "Data Pipeline & Configuration"
        DC1["detection_config.json"]:::config
        SC1["structure_config.json"]:::config
        ENV1["environment.yml"]:::config
    end

    %% Model Training (Detection & Structure Recognition)
    subgraph "Model Training"
        subgraph "Table Detection Training"
            DM1["detr/models/detr.py"]:::train
            DC2["detr/datasets/coco.py"]:::train
            DM2["detr/main.py"]:::train
            DM3["detr/models/detr_multi.py"]:::train
        end
        subgraph "Table Structure Recognition Training"
            DS1["detr/models/segmentation.py"]:::train
            DT1["detr/models/transformer.py"]:::train
            SM1["src/main.py"]:::train
        end
    end

    %% Supporting Utilities
    subgraph "Supporting Utilities"
        U1["detr/util/box_ops.py"]:::utils
        U2["detr/util/misc.py"]:::utils
        U3["detr/util/plot_utils.py"]:::utils
    end

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        IN1["src/inference.py"]:::eval
        EV1["src/eval.py"]:::eval
        TA1["detr/test_all.py"]:::eval
    end

    %% External Integrations & Deployment
    subgraph "External Integrations & Deployment"
        D2["detr/d2"]:::deployment
        GH[".github/workflows"]:::deployment
        DOCK["detr/Dockerfile"]:::deployment
    end

    %% Data Flow Connections
    %% External Data Sources to Data Pre-processing
    HF1 -->|"data"| PF1
    MS1 -->|"data"| PI1
    F1 -->|"data"| PP1
    S1 -->|"data"| PS1
    P1 -->|"data"| VA1

    %% Data Pre-processing to Data Pipeline & Configuration
    VA1 -->|"processed data"| DC1

    %% Configuration to Model Training
    ENV1 -->|"settings"| DM1
    ENV1 -->|"settings"| DS1

    %% Model Training to Inference & Evaluation
    DM2 -->|"trained model"| IN1
    SM1 -->|"trained model"| IN1

    %% Supporting Utilities assist Model Training and Inference
    U1 -.-> DM2
    U2 -.-> IN1
    U3 -.-> EV1

    %% External Integrations supporting Training
    D2 -->|"enhancements"| DM1
    D2 -->|"enhancements"| DS1

    %% Inference to Evaluation
    IN1 -->|"results"| EV1
    IN1 -->|"results"| TA1

    %% CI/CD and Deployment environment connection (indicative)
    GH --> DOCK

    %% Click Events for Data Ingestion & Pre-Processing Scripts
    click PF1 "https://github.com/microsoft/table-transformer/blob/main/scripts/process_fintabnet.py"
    click PI1 "https://github.com/microsoft/table-transformer/blob/main/scripts/process_icdar2013.py"
    click PP1 "https://github.com/microsoft/table-transformer/blob/main/scripts/process_pubmed.py"
    click PS1 "https://github.com/microsoft/table-transformer/blob/main/scripts/process_scitsr.py"
    click VA1 "https://github.com/microsoft/table-transformer/blob/main/scripts/view_annotations.py"

    %% Click Events for Data Pipeline & Configuration
    click DC1 "https://github.com/microsoft/table-transformer/blob/main/src/detection_config.json"
    click SC1 "https://github.com/microsoft/table-transformer/blob/main/src/structure_config.json"
    click ENV1 "https://github.com/microsoft/table-transformer/blob/main/environment.yml"

    %% Click Events for Core Model Architecture & Training (Detection)
    click DM1 "https://github.com/microsoft/table-transformer/blob/main/detr/models/detr.py"
    click DC2 "https://github.com/microsoft/table-transformer/blob/main/detr/datasets/coco.py"
    click DM2 "https://github.com/microsoft/table-transformer/blob/main/detr/main.py"
    click DM3 "https://github.com/microsoft/table-transformer/blob/main/detr/models/detr_multi.py"

    %% Click Events for Core Model Architecture & Training (Structure Recognition)
    click DS1 "https://github.com/microsoft/table-transformer/blob/main/detr/models/segmentation.py"
    click DT1 "https://github.com/microsoft/table-transformer/blob/main/detr/models/transformer.py"
    click SM1 "https://github.com/microsoft/table-transformer/blob/main/src/main.py"

    %% Click Events for Inference & Evaluation
    click IN1 "https://github.com/microsoft/table-transformer/blob/main/src/inference.py"
    click EV1 "https://github.com/microsoft/table-transformer/blob/main/src/eval.py"
    click TA1 "https://github.com/microsoft/table-transformer/blob/main/detr/test_all.py"

    %% Click Events for Supporting Utilities
    click U1 "https://github.com/microsoft/table-transformer/blob/main/detr/util/box_ops.py"
    click U2 "https://github.com/microsoft/table-transformer/blob/main/detr/util/misc.py"
    click U3 "https://github.com/microsoft/table-transformer/blob/main/detr/util/plot_utils.py"

    %% Click Events for External Integrations & Deployment
    click D2 "https://github.com/microsoft/table-transformer/tree/main/detr/d2"
    click GH "https://github.com/microsoft/table-transformer/tree/main/.github/workflows"
    click DOCK "https://github.com/microsoft/table-transformer/tree/main/detr/Dockerfile"

%% Styles
classDef external fill:#ffe0b3, stroke:#d35400, stroke-width:2px;
classDef process fill:#d6eaf8, stroke:#2e86c1, stroke-width:2px;
classDef config fill:#d5f5e3, stroke:#27ae60, stroke-width:2px;
classDef train fill:#f9e79f, stroke:#f39c12, stroke-width:2px;
classDef eval fill:#d1f2eb, stroke:#16a085, stroke-width:2px;
classDef utils fill:#fadbd8, stroke:#e74c3c, stroke-width:2px;
classDef deployment fill:#f5b7b1, stroke:#c0392b, stroke-width:2px;
```
