# mlcommons/training architecture

> Architecture diagram of the GitHub repository mlcommons/training, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mlcommons/training
- Repository: https://github.com/mlcommons/training

## Overview

##### Major Components

– Language Models (BERT, GPT3, LLama2 70B-LoRA)
– Vision and Image Processing (Stable Diffusion, Single-Stage Detector, etc.)
– Graph Neural Networks (e.g., RGAT implemented in “graph_neural_network”)
– Recommendation (e.g., DLRMv2 implemented in “recommendation_v2/torchrec_dlrm”)
– (Additional benchmarks, retired benchmarks, etc. can be indicated as legacy or alternate paths.)

- Shared Infrastructure and Tools:
– Docker Setup: Almost every benchmark folder contains its own Dockerfiles (and in some cases Docker-compose or related build scripts). Represent this as a common “Containerization” component that ensures reproducibility and environment isolation.
– Data Download & Verification: There are several “download_dataset.sh” and “verify_dataset.sh” scripts in many subdirectories. Illustrate these as part of the “Data Ingestion” or “Dataset Management” sub-system connected to each benchmark.
– Logging/MLPerf Utilities: A set of scripts such as “mlperf_logging_utils.py” appear in multiple folders. Represent these common logging and metrics components as a shared service or library.
– CI/CD and Developer Operations: The “.github” directory in the root indicates usage of workflows (like “cla.yml”), and tools like “install_cuda_docker.sh” ensure system setup. Add these as supporting infrastructure.

##### Interactions and Relationships

- Data Flow: For each benchmark, illustrate how:
– The dataset is first downloaded and verified.
– The Docker container is built (using a dedicated Dockerfile) and then the training process is triggered (via bash scripts or pipelined commands).
– Model training, logging, and measurement are performed, with outcomes (e.g., timing results) printed at the end.

- Layering: Think of the system in three main layers:
– Infrastructure Layer: Contains container orchestration, dependency installation (scripts like “install_cuda_docker.sh”) and CI/CD pipelines.
– Application/Benchmark Layer: This includes the actual training implementations (organized by benchmark category). For instance, the “language_model/tensorflow/bert” node would further capture elements like data pre-processing, model definition (in “modeling.py”, “optimization.py”), and training runner scripts (“run_pretraining.py”).
– Logging & Metric Layer: Utilizes common MLPerf logging utilities that capture performance metrics.

- Dependencies: Use directional flows to mark that each benchmark’s training process depends on data ingestion, environment setup (Docker), and then reports metrics (logging). A “common” module (or shared libraries/services) can be drawn as a side-car component that interacts with multiple benchmarks.

##### Architectural Patterns and Technologies

- Microservice/Modular Design: Emphasize that each benchmark is an independent module that can be built, run, and tested separately. Use containers (Docker components) to encapsulate the environment for each benchmark.
- Framework Diversity: Note the difference in frameworks – some benchmarks are implemented using TensorFlow (e.g., BERT), and others with PyTorch (e.g., Stable Diffusion, LLama2, recommendation models). Clearly annotate these nodes with the underlying frameworks or libraries.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        CD["CI/CD & DevOps"]:::shared
        DI["Data Ingestion / Dataset Management"]:::shared
        CON["Containerization (Docker)"]:::shared
    end

    %% Application/Benchmark Layer
    subgraph "Application/Benchmark Layer"
        LM["Language Models: BERT"]:::benchmark_blue
        SD["Vision: Stable Diffusion"]:::benchmark_green
        SSD["Vision: Single-Stage Detector"]:::benchmark_green
        GNN["Graph Neural Networks: RGAT"]:::benchmark_purple
        REC["Recommendation: DLRMv2"]:::benchmark_orange
    end

    %% Logging & Metric Layer
    subgraph "Logging & Metric Layer"
        LOG["MLPerf Logging & Metrics"]:::shared
    end

    %% Infrastructure Relationships
    CD --> DI
    CD --> CON
    DI --> CON

    %% Data flow from Infrastructure to Benchmarks
    CON --> LM
    CON --> SD
    CON --> SSD
    CON --> GNN
    CON --> REC

    DI --> LM
    DI --> SD
    DI --> SSD
    DI --> GNN
    DI --> REC

    %% Benchmark to Logging flow
    LM --> LOG
    SD --> LOG
    SSD --> LOG
    GNN --> LOG
    REC --> LOG

    %% Click Events
    click LM "https://github.com/mlcommons/training/tree/master/language_model/tensorflow/bert"
    click SD "https://github.com/mlcommons/training/tree/master/stable_diffusion"
    click SSD "https://github.com/mlcommons/training/tree/master/single_stage_detector"
    click GNN "https://github.com/mlcommons/training/tree/master/graph_neural_network"
    click REC "https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm"
    click CON "https://github.com/mlcommons/training/tree/master/graph_neural_network/Dockerfile"
    click DI "https://github.com/mlcommons/training/blob/master/graph_neural_network/download_igbh_full.sh"
    click LOG "https://github.com/mlcommons/training/blob/master/stable_diffusion/mlperf_logging_utils.py"
    click CD "https://github.com/mlcommons/training/blob/master/.github/workflows/cla.yml"

    %% Styles
    classDef benchmark_blue fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef benchmark_green fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef benchmark_purple fill:#dda0dd,stroke:#333,stroke-width:2px;
    classDef benchmark_orange fill:#ffa500,stroke:#333,stroke-width:2px;
    classDef shared fill:#f0e68c,stroke:#333,stroke-width:2px;
```
