# clovaai/donut architecture

> Architecture diagram of the GitHub repository clovaai/donut, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/clovaai/donut
- Repository: https://github.com/clovaai/donut

## Overview

##### Project’s Main Components
- Core Model Module (Donut):
- “donut” package (donut/model.py, donut/util.py, and related helper files) as the heart of the document understanding transformer.
- Role in processing document images and producing structured JSON outputs.
- Training & Inference Pipeline:
- “train.py” and “test.py” as entry points for model training and evaluation/inference.
- Use of “lightning_module.py” which indicates that the training loop is built on PyTorch Lightning.
- Use of multiple configuration files from the “config” folder (e.g., train_cord.yaml) to drive experiments.
- Data & Dataset Management:
- “dataset” directory containing the image and metadata (JSON Lines formatted) as the source for training and testing.
- How the ground truth’s JSON structure (gt_parse and gt_parses) is fed into the model.
- Synthetic Data Generation (SynthDoG):
- “synthdog” directory representing the SynthDoG module.
- Components (elements, layouts, resources, and templates) that generate synthetic documents to augment model pre-training.
- Demo / Application:
- Use the “app.py” as the demo application entry point (with Gradio and/or Colab demos) to show model inference in action.

##### Out the Component Interactions
- Data Flow:
- Model Training:
- “train.py” to “lightning_module.py” and further to the “donut” module which encapsulates the Transformer-based document understanding logic.
- How configuration files in “config” are used to set training parameters.
- Inference Flow:
- From the trained model (saved in the “result” directory), show how “test.py” loads the model for inference and evaluation.
- That “app.py” can also load a trained model to serve a real-time demo.
- Synthetic Data Generation Flow:
- From the “synthdog” directory’s elements, layouts, and resources, illustrate a branch that generates synthetic datasets.
- This synthetic data flow back to the training component, where it augments or pre-trains the Donut model.

##### Architectural Patterns and Design Principles
- Modular Design:
- Separation between the training/inference pipelines (train.py, test.py, lightning_module.py) and model definition (donut package).
- Clearly separate the synthetic document generation module (synthdog) from the main model, yet indicate its role in data augmentation.
- Configuration-Driven Workflow:
- “config” files (YAML) as the source of parameters for multiple tasks (CORD, DocVQA, RVL-CDIP, and Train Ticket tasks).
- API/Demo Integration:
- How the demo application (app.py) is a lightweight wrapper that invokes the core model for prediction, reinforcing an API-oriented design.

##### Incorporate Relevant Technologies and Frameworks
- Technologies to label:
- PyTorch for deep learning.
- PyTorch Lightning (lightning_module.py) for training abstraction.
- Hugging Face Transformers (mentioned in the README) for model loading and weight management.
- External libraries such as timm and torchvision.
- Labels near or within the “donut” and training components to give context on dependencies and runtime environment.

- Rectangles for software modules (donut, synthdog, train.py, test.py, app.py).

##### Summary
- Flow into the “Processing Layer” (train.py, lightning_module.py, donut model).
- Then to the “Deployment/Inference Layer” (test.py and app.py for demos).
- With configuration management and external dependencies (PyTorch, Hugging Face, etc.).

Be as detailed as possible and ensure that every module, configuration, and data flow is clearly labeled – this will greatly benefit any further development, maintenance, or feature extension of the project.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Data Layer
  subgraph "Data Layer"
    DS["Dataset (Images & JSON Metadata)"]:::dataLayer
    SD["SynthDoG (Synthetic Data Generator)"]:::dataLayer
  end

  %% Processing / Training Layer
  subgraph "Processing Layer"
    CFG["Config Files"]:::processingLayer
    TR["train.py"]:::processingLayer
    LM["lightning_module.py"]:::processingLayer
    DM["Donut Model"]:::processingLayer
  end

  %% Inference / Demo Layer
  subgraph "Inference / Demo Layer"
    TS["test.py"]:::inferenceLayer
    AP["app.py"]:::inferenceLayer
    MS["Miscellaneous Assets"]:::inferenceLayer
  end

  %% External Dependencies
  subgraph "External Dependencies"
    PT["PyTorch"]:::external
    HF["HuggingFace Transformers"]:::external
    LT["timm & torchvision"]:::external
  end

  %% Data Flow Connections
  DS -->|"feeds_input"| TR
  DS -->|"feeds_input"| TS
  SD -->|"augments_training"| TR

  %% Configuration Flow
  CFG -->|"drives_training"| TR

  %% Training Pipeline Flow
  TR -->|"calls_lightning"| LM
  LM -->|"invokes_model"| DM

  %% Inference Flow
  DM -->|"produces_output"| TS
  DM -->|"serves_demo"| AP
  MS -->|"supports_demo"| AP

  %% External Dependencies Connections
  PT ---|"supports"| TR
  PT ---|"supports"| LM
  HF ---|"integrates_with"| DM
  LT ---|"enhances"| DM

  %% Click Events
  click DS "https://github.com/clovaai/donut/tree/master/dataset/"
  click SD "https://github.com/clovaai/donut/tree/master/synthdog/"
  click CFG "https://github.com/clovaai/donut/tree/master/config/"
  click TR "https://github.com/clovaai/donut/blob/master/train.py"
  click LM "https://github.com/clovaai/donut/blob/master/lightning_module.py"
  click DM "https://github.com/clovaai/donut/blob/master/donut/model.py"
  click TS "https://github.com/clovaai/donut/blob/master/test.py"
  click AP "https://github.com/clovaai/donut/blob/master/app.py"
  click MS "https://github.com/clovaai/donut/tree/master/misc/"

  %% Styles
  classDef dataLayer fill:#a8d5e2,stroke:#333,stroke-width:2px;
  classDef processingLayer fill:#fcebb6,stroke:#333,stroke-width:2px;
  classDef inferenceLayer fill:#c0ffd8,stroke:#333,stroke-width:2px;
  classDef external fill:#ffcccb,stroke:#333,stroke-width:2px;
```
