How ai4bharat/nemo works
NeMo is a PyTorch generative-AI framework whose documented areas include ASR, LLMs, multimodal models, and TTS; this repository also describes an ASR multi-softmax implementation. A typical user configures and runs a model, which consumes task data and produces a model result, with checkpointing and experiment logging supporting that work. The graph preserves those product areas and the sampled image-generation path. Source excerpts are limited, so several unsampled subsystem links are omitted rather than inferred.
Main entry points: nemo NeMo framework · pipeline.py Image inference · stablediffusion_trt_pipeline.py TensorRT guidance · modelPT.py Model base
Components 25
Speech and ASR
- ASR data
nemo/collections/asr/data - Manifest collections
nemo/collections/common/parts/preprocessing/collections.py - ASR models
nemo/collections/asr/models - ASR losses
nemo/collections/asr/losses - ASR metrics
nemo/collections/asr/metrics - Speech simulation
nemo/collections/asr/data/data_simulation.py
Multimodal generation
- Image inference
nemo/collections/multimodal/parts/stable_diffusion/pipeline.py - Diffusion model
nemo/collections/multimodal/models/text_to_image/stable_diffusion/ldm/ddpm.py - Diffusion samplers
nemo/collections/multimodal/models/text_to_image/stable_diffusion/samplers - NeRF text guidance
nemo/collections/multimodal/modules/nerf/guidance/txt2img_guidance_base.py - TensorRT guidance
nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py - TensorRT engine
nemo/collections/multimodal/modules/nerf/utils/trt_engine.py - Multimodal model loading
nemo/collections/multimodal/parts/utils.py
Language and retrieval
- NLP models
nemo/collections/nlp/models - Retrieval service
nemo/collections/nlp/modules/common/megatron/retrieval_services/retrieval_service.py - Distributed NLP runtime
nemo/collections/nlp/parts/nlp_overrides.py
Model runtime
- NeMo framework
nemo - Model base
nemo/core/classes/modelPT.py - Trainer configuration
nemo/core/config/pytorch_lightning.py - Checkpoint connector
nemo/core/connectors/save_restore_connector.py - Experiment manager
nemo/utils/exp_manager.py - Model utilities
nemo/utils/model_utils.py - Rank utilities
nemo/utils/get_rank.py - Common preprocessing
nemo/collections/common/parts/preprocessing/collections.py
Other components
- Researcher
Connections 16
- Researcher → NeMo framework: uses
- NeMo framework → ASR data: configures
- ASR data → Manifest collections: uses collections
nemo/collections/common/parts/preprocessing/collections.py - NeMo framework → ASR models: runs ASR
- NeMo framework → NLP models: runs LLMs
nemo/export/quantize/quantizer.py - NeMo framework → Image inference: runs generation
nemo/collections/multimodal/parts/stable_diffusion/pipeline.py - Image inference → Diffusion model: uses model
- Image inference → Diffusion samplers: selects sampler
nemo/collections/multimodal/parts/stable_diffusion/pipeline.py - TensorRT guidance → NeRF text guidance: extends
nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py - TensorRT guidance → TensorRT engine: uses engine
nemo/collections/multimodal/modules/nerf/guidance/stablediffusion_trt_pipeline.py - Retrieval service → NLP models: supports retrieval
- Model base → Checkpoint connector: saves and restores
nemo/core/classes/modelPT.py - Model base → Model utilities: uses utilities
nemo/core/classes/modelPT.py - Experiment manager → Rank utilities: checks rank
nemo/utils/exp_manager.py - Speech simulation → Common preprocessing: uses collections
- Multimodal model loading → Distributed NLP runtime: configures runtime
nemo/collections/multimodal/parts/utils.py