# doanva84/nemo1 architecture

> Architecture diagram of the GitHub repository doanva84/nemo1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/doanva84/nemo1
- Repository: https://github.com/doanva84/nemo1

## Overview

##### Main Components
- Core Library (“nemo/core”)
– Central classes (ModelPT, Dataset, Exportable, Lightning integration, neural types, connectors).
– Python-based configuration (Hydra-Runner, base_config).
- Domain Collections (“nemo/collections”)
– ASR, TTS, NLP/LLM, VLM/multimodal, Audio, Diffusion, Vision, etc.
– Each collection: data modules, models, losses, metrics, modules, parts (submodules + utils).
- Training Engine (“nemo/lightning”)
– PyTorch Lightning overrides, strategies (TP, PP, FSDP, MoE).
- Export & Deployment (“nemo/export” + “nemo/deploy”)
– ONNX, TensorRT-LLM, Triton, FastAPI microservices, Riva integration, VLLM exporter.
- CLI, Examples & Scripts
– “examples/…” for end-user sample code.
– “scripts/…” for data prep, checkpoint conversion, model training, deployment helper.
– “tools/…” for stand-alone utilities (e.g., CTC segmentation, forced aligner).
- Packaging & Containers
– Dockerfiles under “docker/…”, pip packaging (setup.py, pyproject.toml).
- CI/CD & Quality Checks
– GitHub Actions workflows (.github/workflows) for build/test/publish.
– Linters, code formatting, coverage, labeler, version bump bots.
- External Dependencies & Services
– PyTorch, Hydra, Torch-Lightning, NVIDIA Megatron Core, Transformer Engine.
– Cloud/cluster: AWS EKS, Google GKE, Slurm, NGC containers.
– Model Hub: Hugging Face, NVIDIA NGC.

##### Relationships & Data Flow
– User invokes an example script (or CLI command) → loads a Hydra config → Core Library instantiates DataModule + ModelPT → Training Engine schedules across GPUs → Model outputs to disk or memory.
– Export component reads a trained checkpoint → converter (ONNX/TensorRT) → exported artifact.
– Deployment service (FastAPI/Triton) loads exported model → serves inference requests from external clients.
– CI pipelines trigger on git events → run linters/tests → build Docker images → publish Python wheel to PyPI and container to NGC.

##### Architectural Patterns & Principles
– Modular plugin pattern: each “collection” is self-contained but plugs into the Core API.
– Hydra-based configuration: decouples code from config, supports Python-based configs in v2.0.
– Layered architecture:
Layer 1: User Interface (examples, CLI, scripts)
Layer 2: Core & Collections (library code)
Layer 3: Training Engine (Lightning + Megatron)
Layer 4: Export & Deployment
Layer 5: CI/CD & Packaging
– Separation of concerns: data, model, training, export, and deployment all live in separate modules.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Examples & Scripts
    subgraph "Examples & Scripts"
        Examples["examples/..."]:::example
        Scripts["scripts/..."]:::example
    end

    %% Core & Collections
    subgraph "Core Library" 
        Core["nemo/core"]:::core
        CoreClasses["classes"]:::core
        HydraRunner["config/hydra_runner.py"]:::core
        BaseConfig["config/base_config.py"]:::core
        NeuralTypes["neural_types"]:::core
        Connector["connectors/save_restore_connector.py"]:::core
    end

    subgraph "Domain Collections"
        ASR["nemo/collections/asr"]:::core
        TTS["nemo/collections/tts"]:::core
        LLM["nemo/collections/llm"]:::core
        VLM["nemo/collections/vlm"]:::core
        Multimodal["nemo/collections/multimodal"]:::core
        Audio["nemo/collections/audio"]:::core
        Diffusion["nemo/collections/diffusion"]:::core
        Vision["nemo/collections/vision"]:::core
    end

    %% Training Engine
    subgraph "Training Engine"
        Train["nemo/lightning"]:::training
        Strategies["pytorch/strategies"]:::training
        StratLib["_strategy_lib.py"]:::training
    end

    %% Export
    subgraph "Export"
        ONNX["onnx_llm_exporter.py"]:::export
        TRT["tensorrt_llm.py"]:::export
        VLLM["vllm_exporter.py"]:::export
    end

    %% Deployment
    subgraph "Deployment"
        DEPLOYPY["deploy_pytriton.py"]:::deploy
        FASTAPI["service/fastapi_interface_to_pytriton.py"]:::deploy
        TRITON["triton_deployable.py"]:::deploy
    end

    %% CI/CD & Packaging
    subgraph "CI/CD & Packaging"
        GHWF[".github/workflows/*.yml"]:::cicd
        Actions[".github/actions"]:::cicd
        ScriptsGH[".github/scripts"]:::cicd
        DockerCI["docker/Dockerfile.ci"]:::cicd
        DockerExp["docker/Dockerfile.ci.export_deploy"]:::cicd
        DockerSpeech["docker/Dockerfile.speech"]:::cicd
        Setup["setup.py"]:::cicd
        PyProj["pyproject.toml"]:::cicd
        Linters[".flake8/.pylintrc/.pre-commit-config.yaml"]:::cicd
    end

    %% External Services & Dependencies
    subgraph "External Services & Dependencies"
        Cluster["GPU Cluster (AWS EKS, GKE, Slurm)"]:::external
        HF["Hugging Face / NGC"]:::external
        PyTorch["PyTorch, Hydra, Lightning, Megatron, Transformer Engine"]:::external
    end

    %% Connections
    Examples -->|"usage"| Core
    Scripts -->|"usage"| Core
    Core -->|"provides API"| Train
    ASR --> Train
    TTS --> Train
    LLM --> Train
    VLM --> Train
    Multimodal --> Train
    Audio --> Train
    Diffusion --> Train
    Vision --> Train
    Train -->|"export checkpoint"| ONNX
    Train -->|"export checkpoint"| TRT
    Train -->|"export checkpoint"| VLLM
    ONNX --> DEPLOYPY
    TRT --> FASTAPI
    VLLM --> TRITON
    DEPLOYPY --> Cluster
    FASTAPI --> Cluster
    TRITON --> Cluster
    GHWF --> DockerCI
    GHWF --> DockerExp
    GHWF --> DockerSpeech
    GHWF --> Setup
    GHWF --> PyProj
    GHWF --> Linters
    Setup --> HF
    DockerCI --> HF
    DockerExp --> HF

    %% Click Events
    click Core "https://github.com/doanva84/nemo1/tree/main/nemo/core"
    click CoreClasses "https://github.com/doanva84/nemo1/tree/main/nemo/core/classes"
    click HydraRunner "https://github.com/doanva84/nemo1/blob/main/nemo/core/config/hydra_runner.py"
    click BaseConfig "https://github.com/doanva84/nemo1/blob/main/nemo/core/config/base_config.py"
    click NeuralTypes "https://github.com/doanva84/nemo1/tree/main/nemo/core/neural_types"
    click Connector "https://github.com/doanva84/nemo1/blob/main/nemo/core/connectors/save_restore_connector.py"
    click ASR "https://github.com/doanva84/nemo1/tree/main/nemo/collections/asr"
    click TTS "https://github.com/doanva84/nemo1/tree/main/nemo/collections/tts"
    click LLM "https://github.com/doanva84/nemo1/tree/main/nemo/collections/llm"
    click VLM "https://github.com/doanva84/nemo1/tree/main/nemo/collections/vlm"
    click Multimodal "https://github.com/doanva84/nemo1/tree/main/nemo/collections/multimodal"
    click Audio "https://github.com/doanva84/nemo1/tree/main/nemo/collections/audio"
    click Diffusion "https://github.com/doanva84/nemo1/tree/main/nemo/collections/diffusion"
    click Vision "https://github.com/doanva84/nemo1/tree/main/nemo/collections/vision"
    click Train "https://github.com/doanva84/nemo1/tree/main/nemo/lightning"
    click Strategies "https://github.com/doanva84/nemo1/tree/main/nemo/lightning/pytorch/strategies"
    click StratLib "https://github.com/doanva84/nemo1/blob/main/nemo/lightning/_strategy_lib.py"
    click ONNX "https://github.com/doanva84/nemo1/blob/main/nemo/export/onnx_llm_exporter.py"
    click TRT "https://github.com/doanva84/nemo1/blob/main/nemo/export/tensorrt_llm.py"
    click VLLM "https://github.com/doanva84/nemo1/blob/main/nemo/export/vllm_exporter.py"
    click DEPLOYPY "https://github.com/doanva84/nemo1/blob/main/nemo/deploy/deploy_pytriton.py"
    click FASTAPI "https://github.com/doanva84/nemo1/blob/main/nemo/deploy/service/fastapi_interface_to_pytriton.py"
    click TRITON "https://github.com/doanva84/nemo1/blob/main/nemo/deploy/triton_deployable.py"
    click GHWF "https://github.com/doanva84/nemo1/blob/main/.github/workflows/*.yml"
    click Actions "https://github.com/doanva84/nemo1/tree/main/.github/actions"
    click ScriptsGH "https://github.com/doanva84/nemo1/tree/main/.github/scripts"
    click DockerCI "https://github.com/doanva84/nemo1/blob/main/docker/Dockerfile.ci"
    click DockerExp "https://github.com/doanva84/nemo1/blob/main/docker/Dockerfile.ci.export_deploy"
    click DockerSpeech "https://github.com/doanva84/nemo1/blob/main/docker/Dockerfile.speech"
    click Setup "https://github.com/doanva84/nemo1/blob/main/setup.py"
    click PyProj "https://github.com/doanva84/nemo1/blob/main/pyproject.toml"
    click Linters "https://github.com/doanva84/nemo1/blob/main/.flake8"

    %% Styles
    classDef core fill:#D6EAF8,stroke:#2E86C1,color:#1B4F72
    classDef training fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef export fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef deploy fill:#FDEBD0,stroke:#E67E22,color:#873600
    classDef cicd fill:#FCF3CF,stroke:#F1C40F,color:#7D6608
    classDef example fill:#F2F3F4,stroke:#B3B6B7,color:#424949
    classDef external fill:#EBDEF0,stroke:#9B59B6,color:#4A235A
```
