# ai4bharat/nemo architecture

> Architecture diagram of the GitHub repository ai4bharat/nemo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ai4bharat/nemo
- Repository: https://github.com/ai4bharat/nemo

## Overview

NeMo is a PyTorch generative-AI framework whose documented areas include ASR, LLMs, multimodal models, and TTS; this repository also describes an ASR multi-softmax implementation. A typical user configures and runs a model, which consumes task data and produces a model result, with checkpointing and experiment logging supporting that work. The graph preserves those product areas and the sampled image-generation path. Source excerpts are limited, so several unsampled subsystem links are omitted rather than inferred.

Main entry points: [`nemo`](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo) (NeMo framework), [`nemo/collections/multimodal/parts/stable_diffusion/pipeline.py`](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/parts/stable_diffusion/pipeline.py) (Image inference), [`nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py`](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py) (TensorRT guidance), [`nemo/core/classes/modelPT.py`](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/classes/modelPT.py) (Model base)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_asr["Speech and ASR"]
  node_asr_data["ASR data"]
  node_manifest_collections["Manifest collections<br/>[collections.py]"]
  node_asr_models["ASR models"]
  node_asr_losses["ASR losses"]
  node_asr_metrics["ASR metrics"]
  node_asr_simulation["Speech simulation<br/>[data_simulation.py]"]
end

subgraph group_multimodal["Multimodal generation"]
  node_diffusion_pipeline["Image inference<br/>[pipeline.py]"]
  node_diffusion_model["Diffusion model<br/>[ddpm.py]"]
  node_diffusion_samplers["Diffusion samplers"]
  node_nerf_guidance["NeRF text guidance"]
  node_trt_guidance["TensorRT guidance"]
  node_trt_engine["TensorRT engine<br/>[trt_engine.py]"]
  node_multimodal_utils["Multimodal model loading<br/>[utils.py]"]
end

subgraph group_nlp["Language and retrieval"]
  node_nlp_models["NLP models"]
  node_retrieval_service["Retrieval service"]
  node_nlp_overrides["Distributed NLP runtime<br/>[nlp_overrides.py]"]
end

subgraph group_core["Model runtime"]
  node_framework["NeMo framework"]
  node_modelpt["Model base<br/>[modelPT.py]"]
  node_trainer_config["Trainer configuration"]
  node_checkpoint_connector["Checkpoint connector"]
  node_experiment_manager["Experiment manager<br/>[exp_manager.py]"]
  node_model_utils["Model utilities<br/>[model_utils.py]"]
  node_rank_utils["Rank utilities<br/>[get_rank.py]"]
  node_common_preprocessing["Common preprocessing<br/>[collections.py]"]
end

node_researcher(("Researcher"))

node_researcher -->|"uses"| node_framework
node_framework -.->|"configures"| node_asr_data
node_asr_data -->|"uses collections"| node_manifest_collections
node_framework -.->|"runs ASR"| node_asr_models
node_framework -.->|"runs LLMs"| node_nlp_models
node_framework -.->|"runs generation"| node_diffusion_pipeline
node_diffusion_pipeline -.->|"uses model"| node_diffusion_model
node_diffusion_pipeline -->|"selects sampler"| node_diffusion_samplers
node_trt_guidance -->|"extends"| node_nerf_guidance
node_trt_guidance -->|"uses engine"| node_trt_engine
node_retrieval_service -.->|"supports retrieval"| node_nlp_models
node_modelpt -->|"saves and restores"| node_checkpoint_connector
node_modelpt -->|"uses utilities"| node_model_utils
node_experiment_manager -->|"checks rank"| node_rank_utils
node_asr_simulation -.->|"uses collections"| node_common_preprocessing
node_multimodal_utils -->|"configures runtime"| node_nlp_overrides

click node_framework "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo"
click node_asr_data "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/data"
click node_manifest_collections "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/common/parts/preprocessing/collections.py"
click node_asr_models "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/models"
click node_asr_losses "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/losses"
click node_asr_metrics "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/metrics"
click node_asr_simulation "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/asr/data/data_simulation.py"
click node_diffusion_pipeline "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/parts/stable_diffusion/pipeline.py"
click node_diffusion_model "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/models/text_to_image/stable_diffusion/ldm/ddpm.py"
click node_diffusion_samplers "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/multimodal/models/text_to_image/stable_diffusion/samplers"
click node_nerf_guidance "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/guidance/txt2img_guidance_base.py"
click node_trt_guidance "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py"
click node_trt_engine "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/utils/trt_engine.py"
click node_nlp_models "https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/nlp/models"
click node_retrieval_service "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/nlp/modules/common/megatron/retrieval_services/retrieval_service.py"
click node_modelpt "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/classes/modelPT.py"
click node_trainer_config "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/config/pytorch_lightning.py"
click node_checkpoint_connector "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/connectors/save_restore_connector.py"
click node_experiment_manager "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/utils/exp_manager.py"
click node_model_utils "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/utils/model_utils.py"
click node_rank_utils "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/utils/get_rank.py"
click node_common_preprocessing "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/common/parts/preprocessing/collections.py"
click node_nlp_overrides "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/nlp/parts/nlp_overrides.py"
click node_multimodal_utils "https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/parts/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_asr_data,node_manifest_collections,node_asr_models,node_asr_losses,node_asr_metrics,node_asr_simulation toneBlue
class node_diffusion_pipeline,node_diffusion_model,node_diffusion_samplers,node_nerf_guidance,node_trt_guidance,node_trt_engine,node_multimodal_utils toneAmber
class node_nlp_models,node_retrieval_service,node_nlp_overrides toneMint
class node_framework,node_modelpt,node_trainer_config,node_checkpoint_connector,node_experiment_manager,node_model_utils,node_rank_utils,node_common_preprocessing toneRose
class node_researcher toneIndigo
```

## Components

### Speech and ASR

- [ASR data](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/data): `nemo/collections/asr/data`
- [Manifest collections](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/common/parts/preprocessing/collections.py): `nemo/collections/common/parts/preprocessing/collections.py`
- [ASR models](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/models): `nemo/collections/asr/models`
- [ASR losses](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/losses): `nemo/collections/asr/losses`
- [ASR metrics](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/asr/metrics): `nemo/collections/asr/metrics`
- [Speech simulation](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/asr/data/data_simulation.py): `nemo/collections/asr/data/data_simulation.py`

### Multimodal generation

- [Image inference](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/parts/stable_diffusion/pipeline.py): `nemo/collections/multimodal/parts/stable_diffusion/pipeline.py`
- [Diffusion model](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/models/text_to_image/stable_diffusion/ldm/ddpm.py): `nemo/collections/multimodal/models/text_to_image/stable_diffusion/ldm/ddpm.py`
- [Diffusion samplers](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/multimodal/models/text_to_image/stable_diffusion/samplers): `nemo/collections/multimodal/models/text_to_image/stable_diffusion/samplers`
- [NeRF text guidance](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/guidance/txt2img_guidance_base.py): `nemo/collections/multimodal/modules/nerf/guidance/txt2img_guidance_base.py`
- [TensorRT guidance](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py): `nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py`
- [TensorRT engine](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/modules/nerf/utils/trt_engine.py): `nemo/collections/multimodal/modules/nerf/utils/trt_engine.py`
- [Multimodal model loading](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/multimodal/parts/utils.py): `nemo/collections/multimodal/parts/utils.py`

### Language and retrieval

- [NLP models](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo/collections/nlp/models): `nemo/collections/nlp/models`
- [Retrieval service](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/nlp/modules/common/megatron/retrieval_services/retrieval_service.py): `nemo/collections/nlp/modules/common/megatron/retrieval_services/retrieval_service.py`
- [Distributed NLP runtime](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/nlp/parts/nlp_overrides.py): `nemo/collections/nlp/parts/nlp_overrides.py`

### Model runtime

- [NeMo framework](https://github.com/ai4bharat/nemo/tree/nemo-v2/nemo): `nemo`
- [Model base](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/classes/modelPT.py): `nemo/core/classes/modelPT.py`
- [Trainer configuration](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/config/pytorch_lightning.py): `nemo/core/config/pytorch_lightning.py`
- [Checkpoint connector](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/core/connectors/save_restore_connector.py): `nemo/core/connectors/save_restore_connector.py`
- [Experiment manager](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/utils/exp_manager.py): `nemo/utils/exp_manager.py`
- [Model utilities](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/utils/model_utils.py): `nemo/utils/model_utils.py`
- [Rank utilities](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/utils/get_rank.py): `nemo/utils/get_rank.py`
- [Common preprocessing](https://github.com/ai4bharat/nemo/blob/nemo-v2/nemo/collections/common/parts/preprocessing/collections.py): `nemo/collections/common/parts/preprocessing/collections.py`

### Other components

- **Researcher**

## Connections

- Researcher → NeMo framework: uses
- NeMo framework → ASR data: configures
- ASR data → Manifest collections: uses collections (evidence: `nemo/collections/common/parts/preprocessing/collections.py`)
- NeMo framework → ASR models: runs ASR
- NeMo framework → NLP models: runs LLMs (evidence: `nemo/export/quantize/quantizer.py`)
- NeMo framework → Image inference: runs generation (evidence: `nemo/collections/multimodal/parts/stable_diffusion/pipeline.py`)
- Image inference → Diffusion model: uses model
- Image inference → Diffusion samplers: selects sampler (evidence: `nemo/collections/multimodal/parts/stable_diffusion/pipeline.py`)
- TensorRT guidance → NeRF text guidance: extends (evidence: `nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py`)
- TensorRT guidance → TensorRT engine: uses engine (evidence: `nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py`)
- Retrieval service → NLP models: supports retrieval
- Model base → Checkpoint connector: saves and restores (evidence: `nemo/core/classes/modelPT.py`)
- Model base → Model utilities: uses utilities (evidence: `nemo/core/classes/modelPT.py`)
- Experiment manager → Rank utilities: checks rank (evidence: `nemo/utils/exp_manager.py`)
- Speech simulation → Common preprocessing: uses collections
- Multimodal model loading → Distributed NLP runtime: configures runtime (evidence: `nemo/collections/multimodal/parts/utils.py`)
