# nvidia/nemo architecture

> Architecture diagram of the GitHub repository nvidia/nemo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/nemo
- Repository: https://github.com/nvidia/nemo

## Overview

This project is the NVIDIA NeMo Framework – a comprehensive and modular deep learning framework built for training and deploying state‐of‐the‐art models in multiple AI domains (LLMs, multimodal models, ASR, TTS, CV, etc.). The extensive file tree and the detailed README show that the project is organized into “collections” (domain‐specific submodules for ASR, NLP, TTS, vision, multimodal, etc.), a “core” layer (for neural types, configuration, distributed training, and model utilities) and ancillary tooling (for deployment, export, evaluation, testing, and scripts). It also integrates with third‐party libraries such as PyTorch, PyTorch Lightning, NVIDIA TransformerEngine, Megatron Core, and other NVIDIA–specific cloud and container tools (NGC containers, NeMo Run, etc.).

##### Major architectural layers/components
- Domain “Collections”: This is a modular layer that contains domain‐specific implementations. Subdirectories in “nemo/collections” cover ASR, NLP, TTS, vision, multimodal, etc. Each collection encapsulates the models, data modules, losses, metrics, and sometimes tokenizers that are specific to that domain.
- Distributed Training & Scalability: The system leverages advanced parallelism techniques (Tensor/Pipeline/Fully Sharded Data Parallelism, MoE) and distributed training infrastructure (NeMo Run, container orchestration using NGC containers, integration with cloud providers). Represent these as a “Training & Inference Engine” component that interacts with the core and collections.
- Deploy & Export Tools: There is an export and deployment stream (with support for Triton, ONNX, TensorRT, vLLM) found in “nemo/export” and “nemo/deploy”. This handles optimization and serving.
- External Integrations & Ecosystem: Several scripts, tutorials, and integrations (NeMo Curator for video, Cosmos tokenizers and world models, NVIDIA NGC/NGC containers, and third-party platforms like Hugging Face) support end‐to‐end workflows.

##### Cross-cutting architectural concerns
- Modularity: The design stresses modularity; individual models and configurations are “plug‐in”-able and customizable.
- Configurability: A Python-based configuration system governs experiments, training routines, deployment parameters, etc.
- Scalability & Performance: Distributed training utilities, multi‐GPU management, integration with NVIDIA hardware (Hopper GPUs, H200, H100) have bearings on both the architecture and data flow.
- Interoperability: NeMo integrates with external libraries (Megatron, TransformerEngine) and ecosystem services (NGC container, cloud training frameworks).

## Diagram (Mermaid)

```mermaid
flowchart TD
    UI["User Interface & Experiment Management"]:::ui

    subgraph "Core Infrastructure"
        Core1["Core Module"]
        Core2["Lightning Module"]
    end

    subgraph "Domain Collections"
        D1["ASR Collection"]
        D2["NLP Collection"]
        D3["TTS Collection"]
        D4["Vision Collection"]
        D5["Multimodal Collection"]
        D6["LLM Collection"]
    end

    TE["Training & Inference Engine<br/>(FSDP, MoE, TP)"]:::training

    subgraph "Deploy & Export Tools"
        E1["Export Suite"]
        DPL["Deployment Suite"]
    end

    EI["External Integrations & Ecosystem"]:::external

    %% Connections from User Interface
    UI -->|"triggers"| Core1
    UI -->|"triggers"| Core2
    UI -->|"utilizes"| EI

    %% Core Infrastructure supports Domain Collections
    Core1 -->|"provides base"| D1
    Core1 -->|"provides base"| D2
    Core1 -->|"provides base"| D3
    Core1 -->|"provides base"| D4
    Core1 -->|"provides base"| D5
    Core1 -->|"provides base"| D6

    %% Domain Collections and Core feed into Training Engine
    Core2 -->|"integrates with"| TE
    D1 -->|"supplies models"| TE
    D2 -->|"supplies models"| TE
    D3 -->|"supplies models"| TE
    D4 -->|"supplies models"| TE
    D5 -->|"supplies models"| TE
    D6 -->|"supplies models"| TE

    %% Training Engine connects to Deploy & Export Tools
    TE -->|"exports optimized model"| E1
    TE -->|"prepares for serving"| DPL

    %% External Integrations communicates with Training and Deployment
    EI -->|"orchestrates"| TE
    DPL -->|"integrates with"| EI

    %% Click Events for Component Mapping
    click Core1 "https://github.com/nvidia/nemo/tree/main/nemo/core"
    click Core2 "https://github.com/nvidia/nemo/tree/main/nemo/lightning"
    click D1 "https://github.com/nvidia/nemo/tree/main/nemo/collections/asr"
    click D2 "https://github.com/nvidia/nemo/tree/main/nemo/collections/nlp"
    click D3 "https://github.com/nvidia/nemo/tree/main/nemo/collections/tts"
    click D4 "https://github.com/nvidia/nemo/tree/main/nemo/collections/vision"
    click D5 "https://github.com/nvidia/nemo/tree/main/nemo/collections/multimodal"
    click D6 "https://github.com/nvidia/nemo/tree/main/nemo/collections/llm"
    click TE "https://github.com/nvidia/nemo/tree/main/nemo/lightning/pytorch/strategies"
    click E1 "https://github.com/nvidia/nemo/tree/main/nemo/export"
    click DPL "https://github.com/nvidia/nemo/tree/main/nemo/deploy"

    %% Styles
    classDef ui fill:#CCD1D1,stroke:#5D6D7E,stroke-width:2px;
    classDef core fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef collection fill:#ABEBC6,stroke:#1D8348,stroke-width:2px;
    classDef training fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef deploy fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef external fill:#D7BDE2,stroke:#6C3483,stroke-width:2px;

    class Core1,Core2 core;
    class D1,D2,D3,D4,D5,D6 collection;
    class TE training;
    class E1,DPL deploy;
    class EI external;
```
