# nvidia-nemo/automodel architecture

> Architecture diagram of the GitHub repository nvidia-nemo/automodel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia-nemo/automodel
- Repository: https://github.com/nvidia-nemo/automodel

## Overview

##### Project Type & Purpose
- Full-stack, modular Python library and CLI for large-scale training and fine-tuning of LLMs and VLMs.
- Key goals: Day-0 Hugging Face integration, PyTorch DTensor-native SPMD parallelism, recipe-driven workflows, and scalable multi-node deployment (Slurm, Kubernetes).

##### High-Level Structure (from file tree + README)
Top-level directories:
- nemo_automodel/ – core library (CLI, components, recipes, shared utilities)
- examples/ – end-user recipes for LLM/VLM pretrain, finetune, knowledge distillation, generation
- docs/ – Sphinx conf, guides, architecture docs, performance summary
- tests/ – unit + functional test suites organized by component
- docker/, .github/, workflows – CI/CD, containerization, release pipelines

##### Main Components & Their Relationships
a. CLI Launcher (“automodel”)
– Path: nemo_automodel/_cli/app.py
– Parses CLI args, selects recipe (LLM or VLM), loads YAML config via Config Loader, and invokes the main training loop.

b. Recipe Layer
– Path: nemo_automodel/recipes/llm and /vlm
– Implements the domain-specific “train loop” (pretrain, SFT, PEFT, KD).
– Depends on core components: datasets, models, distributed, loss, optim, quantization, checkpointing, loggers.

c. Core Component Library (nemo_automodel/components)
- Config Loader – parse YAML/CLI overrides → builds Python dataclass config.
- Datasets – LLM and VLM dataset builders, column-mapped, packers.
- Models – wrappers around Hugging Face Transformers / Diffusers via registry (auto_model).
- Distributed – DTensor mesh setup, FSDP2/HSDP, data/tensor/sequence/pipeline parallelism, Megatron bridge.
- MoE – expert token dispatchers, custom MoE kernels.
- Loss – optimized cross-entropy, KD, Triton kernels.
- Optim – scheduler, FP8 quantization modules.
- Checkpointing – distributed SafeTensors shards, merge/reshard utilities.
- Launcher – Slurm script generator, K8s job spec (in docker/), universal “torchrun” wrapper.
- Loggers – metric logging, Weights & Biases integration.
- Utils – compile/jit helpers, flops, YAML, signature utils.

d. External Dependencies & Infrastructure
- PyTorch (DTensor, torch.compile, torchrun)
- Huggingface Hub (model/tokenizer I/O)
- Storage (local FS, network-attached, S3, HF cache)
- Cluster Manager – Slurm or Kubernetes (via job scripts)
- Container image (docker/Dockerfile, uv lockfiles)
- CI/CD – GitHub Actions, codecov, release workflows

##### Architectural Patterns & Design Principles
- Modular, hackable plugin architecture (registry pattern for models, datasets).
- Recipe-driven orchestration: decoupled “what to run” (YAML) vs. “how to parallelize” (DTensor placement).
- Single code-path for any scale (1 GPU to thousands via mesh config).
- Separation of concerns: CLI, config, data, model, distributed strategy, checkpointing, logging.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layer 1: User Interface
    cli["CLI Launcher"]:::ui
    examples["Examples"]:::ui

    %% Layer 2: Recipe Engine
    subgraph "Recipe Engine"
        direction TB
        llm_ft["LLM Finetune (train_ft.py)"]:::recipe
        llm_bench["LLM Benchmark (benchmark.py)"]:::recipe
        llm_kd["LLM KD (kd.py)"]:::recipe
        vlm_ft["VLM Finetune (finetune.py)"]:::recipe
    end

    %% Layer 3: Core Components
    subgraph "Core Components"
        direction TB
        subgraph Config
            config_loader["Config Loader"]:::core
            arg_parser["Arg Parser"]:::core
        end
        subgraph Datasets
            llm_data["LLM Datasets"]:::core
            vlm_data["VLM Datasets"]:::core
        end
        models["Model Wrappers"]:::core
        distributed["Distributed Engine"]:::core
        moe["Mixture of Experts"]:::core
        loss["Loss Modules"]:::core
        optim["Optim & Scheduler"]:::core
        checkpoint["Checkpointing"]:::core
        launcher["Launcher Scripts"]:::core
        loggers["Loggers"]:::core
        utils["Utility Modules"]:::core
        metadata["Package Metadata"]:::core
    end

    %% Layer 4: External Systems
    subgraph "External Systems & Infra"
        direction TB
        pytorch["PyTorch Runtime"]:::external
        hf["Hugging Face Hub"]:::external
        storage["Storage (FS/S3)"]:::external
        cluster["Cluster Manager (Slurm/K8s)"]:::external
        docker["Docker Container"]:::external
        cicd["CI/CD Pipelines"]:::external
    end

    %% Connections
    cli -->|"launches"| config_loader
    cli -->|"selects recipe"| llm_ft
    cli -->|"selects recipe"| llm_bench
    cli -->|"selects recipe"| llm_kd
    cli -->|"selects recipe"| vlm_ft
    examples -->|"provide recipes"| llm_ft
    examples -->|"provide recipes"| vlm_ft

    llm_ft -->|uses| llm_data
    llm_bench -->|uses| llm_data
    llm_kd -->|uses| llm_data
    vlm_ft -->|uses| vlm_data

    llm_ft -->|calls| models
    llm_bench -->|calls| models
    llm_kd -->|calls| models
    vlm_ft -->|calls| models

    llm_ft -->|parallelism| distributed
    llm_bench -->|parallelism| distributed
    llm_kd -->|parallelism| distributed
    vlm_ft -->|parallelism| distributed

    llm_ft -->|moe| moe
    llm_kd -->|moe| moe

    llm_ft -->|compute loss| loss
    llm_kd -->|compute loss| loss
    vlm_ft -->|compute loss| loss

    llm_ft -->|optim| optim
    llm_kd -->|optim| optim
    vlm_ft -->|optim| optim

    llm_ft -->|save| checkpoint
    llm_bench -->|save| checkpoint
    llm_kd -->|save| checkpoint

    llm_ft -->|launch jobs| launcher
    vlm_ft -->|launch jobs| launcher

    llm_ft -->|log metrics| loggers
    vlm_ft -->|log metrics| loggers

    llm_ft -->|utils| utils
    llm_bench -->|utils| utils
    llm_kd -->|utils| utils
    vlm_ft -->|utils| utils

    cli -->|metadata| metadata

    %% Core to External
    models -->|loads from| hf
    models -->|diffusion via| hf
    llm_data -->|read/write| storage
    vlm_data -->|read/write| storage
    checkpoint -->|store shards| storage

    distributed -->|runtime| pytorch
    moe -->|kernels| pytorch
    loss -->|kernels| pytorch
    optim -->|quant| pytorch

    launcher -->|submit jobs| cluster
    launcher -->|container| docker

    cicd -->|build| docker
    cicd -->|test| cli

    %% Click Events
    click cli "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/_cli/app.py"
    click llm_ft "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/recipes/llm/train_ft.py"
    click llm_bench "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/recipes/llm/benchmark.py"
    click llm_kd "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/recipes/llm/kd.py"
    click vlm_ft "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/recipes/vlm/finetune.py"
    click config_loader "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/config/loader.py"
    click arg_parser "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/config/_arg_parser.py"
    click llm_data "https://github.com/nvidia-nemo/automodel/tree/main/nemo_automodel/components/datasets/llm/"
    click vlm_data "https://github.com/nvidia-nemo/automodel/tree/main/nemo_automodel/components/datasets/vlm/"
    click models "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/_transformers/auto_model.py"
    click distributed "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/distributed/init_utils.py"
    click moe "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/moe/layers.py"
    click loss "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/loss/chunked_ce.py"
    click optim "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/optim/scheduler.py"
    click checkpoint "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/checkpoint/checkpointing.py"
    click launcher "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/launcher/slurm/config.py"
    click loggers "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/loggers/metric_logger.py"
    click utils "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/components/utils/compile_utils.py"
    click metadata "https://github.com/nvidia-nemo/automodel/blob/main/nemo_automodel/package_info.py"
    click examples "https://github.com/nvidia-nemo/automodel/tree/main/examples/"

    %% Styles
    classDef ui fill:#D0E8FF,stroke:#0366D6,stroke-width:1px
    classDef recipe fill:#FFD8A8,stroke:#D97504,stroke-width:1px
    classDef core fill:#D6F5D6,stroke:#028A02,stroke-width:1px
    classDef external fill:#E0E0E0,stroke:#888888,stroke-width:1px
```
