How clovaai/donut works
Project’s Main Components
• Core Model Module (Donut):
• “donut” package (donut/model.py, donut/util.py, and related helper files) as the heart of the document understanding transformer.
• Role in processing document images and producing structured JSON outputs.
• Training & Inference Pipeline:
• “train.py” and “test.py” as entry points for model training and evaluation/inference.
• Use of “lightning_module.py” which indicates that the training loop is built on PyTorch Lightning.
• Use of multiple configuration files from the “config” folder (e.g., train_cord.yaml) to drive experiments.
• Data & Dataset Management:
• “dataset” directory containing the image and metadata (JSON Lines formatted) as the source for training and testing.
• How the ground truth’s JSON structure (gt_parse and gt_parses) is fed into the model.
• Synthetic Data Generation (SynthDoG):
• “synthdog” directory representing the SynthDoG module.
• Components (elements, layouts, resources, and templates) that generate synthetic documents to augment model pre-training.
• Demo / Application:
• Use the “app.py” as the demo application entry point (with Gradio and/or Colab demos) to show model inference in action.
Out the Component Interactions
• Data Flow:
• Model Training:
• “train.py” to “lightning_module.py” and further to the “donut” module which encapsulates the Transformer-based document understanding logic.
• How configuration files in “config” are used to set training parameters.
• Inference Flow:
• From the trained model (saved in the “result” directory), show how “test.py” loads the model for inference and evaluation.
• That “app.py” can also load a trained model to serve a real-time demo.
• Synthetic Data Generation Flow:
• From the “synthdog” directory’s elements, layouts, and resources, illustrate a branch that generates synthetic datasets.
• This synthetic data flow back to the training component, where it augments or pre-trains the Donut model.
Architectural Patterns and Design Principles
• Modular Design:
• Separation between the training/inference pipelines (train.py, test.py, lightning_module.py) and model definition (donut package).
• Clearly separate the synthetic document generation module (synthdog) from the main model, yet indicate its role in data augmentation.
• Configuration-Driven Workflow:
• “config” files (YAML) as the source of parameters for multiple tasks (CORD, DocVQA, RVL-CDIP, and Train Ticket tasks).
• API/Demo Integration:
• How the demo application (app.py) is a lightweight wrapper that invokes the core model for prediction, reinforcing an API-oriented design.
Incorporate Relevant Technologies and Frameworks
• Technologies to label:
• PyTorch for deep learning.
• PyTorch Lightning (lightning_module.py) for training abstraction.
• Hugging Face Transformers (mentioned in the README) for model loading and weight management.
• External libraries such as timm and torchvision.
• Labels near or within the “donut” and training components to give context on dependencies and runtime environment.
• Rectangles for software modules (donut, synthdog, train.py, test.py, app.py).
Summary
• Flow into the “Processing Layer” (train.py, lightning_module.py, donut model).
• Then to the “Deployment/Inference Layer” (test.py and app.py for demos).
• With configuration management and external dependencies (PyTorch, Hugging Face, etc.).
Be as detailed as possible and ensure that every module, configuration, and data flow is clearly labeled – this will greatly benefit any further development, maintenance, or feature extension of the project.