# nvidia-nemo/nemo architecture

> Architecture diagram of the GitHub repository nvidia-nemo/nemo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia-nemo/nemo
- Repository: https://github.com/nvidia-nemo/nemo
- Video tour (about a minute): https://gitdiagram.com/nvidia-nemo/nemo/video

## Overview

NeMo Speech is a PyTorch toolkit for building, adapting, and deploying speech and audio models. The principal workflow shown here is configured streaming ASR: a caller selects a pipeline, the builder constructs a buffered or cache-aware pipeline, and streaming requests are processed into transcripts with optional inverse text normalization and translation. The repository also contains training/data foundations, audio-generation and enhancement capabilities, and a real-time voice-agent integration. The README establishes these broader capabilities; sampled source excerpts are concentrated on ASR inference, so uncertain wiring among other subsystems is intentionally omitted.

Main entry points: [`nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py`](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py) (Buffered pipeline), [`nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py`](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py) (Cache-aware pipeline), [`nemo/agents/voice_agent/pipecat/frames/frames.py`](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/frames/frames.py) (Conversation frames), [`nemo/collections/asr/inference/factory/pipeline_builder.py`](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/pipeline_builder.py) (Pipeline router)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_asr["Streaming ASR"]
  node_pipeline_builder["Pipeline router"]
  node_buffer_builder["Buffered builder"]
  node_cache_builder["Cache-aware builder"]
  node_buffered_pipeline["Buffered pipeline"]
  node_cache_pipeline["Cache-aware pipeline"]
  node_base_pipeline["Pipeline base<br/>[base_pipeline.py]"]
  node_stream_state["Stream state<br/>[state.py]"]
  node_asr_wrappers["Inference wrappers"]
  node_itn["Text normalization"]
  node_translation["Speech translation<br/>[llm_translator.py]"]
end

subgraph group_models["Speech Models"]
  node_asr_models["ASR models<br/>[asr_model.py]"]
  node_asr_modules["Encoders and decoders"]
  node_asr_losses["ASR objectives<br/>[ctc.py]"]
  node_asr_metrics["ASR metrics<br/>[wer.py]"]
end

subgraph group_voice["Voice Agent"]
  node_voice_transport["WebSocket transport"]
  node_voice_config["Agent configuration<br/>[config_manager.py]"]
  node_voice_asr["Streaming recognition<br/>[streaming_asr.py]"]
  node_voice_llm["Conversation LLM<br/>[llm.py]"]
  node_voice_tts["Speech synthesis<br/>[tts.py]"]
  node_voice_frames["Conversation frames<br/>[frames.py]"]
end

subgraph group_audio["Audio Generation"]
  node_audio_models["Audio transformation<br/>[audio_to_audio.py]"]
  node_audio_data["Audio datasets<br/>[audio_to_audio.py]"]
  node_magpie_pipeline["Magpie TTS stages<br/>[pipeline.py]"]
end

subgraph group_data["Data and Training"]
  node_speech_data["Speech datasets<br/>[audio_to_text.py]"]
  node_common_data["Shared data loading<br/>[dataloader.py]"]
  node_experiment_manager["Experiment management<br/>[exp_manager.py]"]
end

node_caller(("Speech client"))
node_transcript["Transcript"]
node_audio_user(("Audio user"))

node_caller -->|"selects pipeline"| node_pipeline_builder
node_pipeline_builder -->|"builds buffered"| node_buffer_builder
node_pipeline_builder -->|"builds cache-aware"| node_cache_builder
node_buffer_builder -->|"constructs"| node_buffered_pipeline
node_cache_builder -->|"constructs"| node_cache_pipeline
node_buffered_pipeline -->|"uses base"| node_base_pipeline
node_cache_pipeline -->|"uses base"| node_base_pipeline
node_caller -->|"streams audio"| node_buffered_pipeline
node_caller -->|"streams audio"| node_cache_pipeline
node_buffered_pipeline -->|"runs inference"| node_asr_wrappers
node_cache_pipeline -->|"runs inference"| node_asr_wrappers
node_asr_wrappers -->|"invokes model"| node_asr_models
node_asr_models -->|"uses components"| node_asr_modules
node_base_pipeline -->|"tracks streams"| node_stream_state
node_buffered_pipeline -.->|"normalizes text"| node_itn
node_buffered_pipeline -.->|"translates text"| node_translation
node_cache_pipeline -.->|"normalizes text"| node_itn
node_cache_pipeline -.->|"translates text"| node_translation
node_buffered_pipeline -->|"produces transcript"| node_transcript
node_cache_pipeline -->|"produces transcript"| node_transcript
node_speech_data -->|"uses loading"| node_common_data
node_audio_data -->|"uses loading"| node_common_data
node_voice_transport -->|"dispatches audio"| node_voice_asr
node_voice_config -->|"configures"| node_voice_asr
node_voice_asr -->|"emits recognition"| node_voice_frames
node_voice_frames -->|"supplies dialogue"| node_voice_llm
node_voice_llm -->|"emits response"| node_voice_frames
node_voice_frames -->|"supplies text"| node_voice_tts
node_voice_tts -->|"returns speech"| node_audio_user
node_audio_user -->|"provides audio"| node_audio_models
node_audio_models -->|"returns audio"| node_audio_user
node_magpie_pipeline -->|"generates speech"| node_audio_user
node_experiment_manager -.->|"supports experiments"| node_asr_models

click node_pipeline_builder "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/pipeline_builder.py"
click node_buffer_builder "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/buffered_pipeline_builder.py"
click node_cache_builder "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py"
click node_buffered_pipeline "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py"
click node_cache_pipeline "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py"
click node_base_pipeline "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/base_pipeline.py"
click node_stream_state "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/streaming/state/state.py"
click node_asr_wrappers "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py"
click node_asr_models "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/models/asr_model.py"
click node_asr_modules "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/modules/conformer_encoder.py"
click node_speech_data "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/data/audio_to_text.py"
click node_common_data "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/common/data/lhotse/dataloader.py"
click node_asr_losses "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/losses/ctc.py"
click node_asr_metrics "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/metrics/wer.py"
click node_itn "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/itn/inverse_normalizer.py"
click node_translation "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/nmt/llm_translator.py"
click node_voice_transport "https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py"
click node_voice_config "https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/utils/config_manager.py"
click node_voice_asr "https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/services/nemo/streaming_asr.py"
click node_voice_llm "https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/services/nemo/llm.py"
click node_voice_tts "https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/services/nemo/tts.py"
click node_voice_frames "https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/frames/frames.py"
click node_audio_models "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/audio/models/audio_to_audio.py"
click node_audio_data "https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/audio/data/audio_to_audio.py"
click node_magpie_pipeline "https://github.com/nvidia-nemo/nemo/blob/main/tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py"
click node_experiment_manager "https://github.com/nvidia-nemo/nemo/blob/main/nemo/utils/exp_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_builder,node_buffer_builder,node_cache_builder,node_buffered_pipeline,node_cache_pipeline,node_base_pipeline,node_stream_state,node_asr_wrappers,node_itn,node_translation,node_caller,node_audio_user toneBlue
class node_asr_models,node_asr_modules,node_asr_losses,node_asr_metrics toneAmber
class node_voice_transport,node_voice_config,node_voice_asr,node_voice_llm,node_voice_tts,node_voice_frames toneMint
class node_audio_models,node_audio_data,node_magpie_pipeline toneRose
class node_speech_data,node_common_data,node_experiment_manager,node_transcript toneIndigo
```

## Components

### Streaming ASR

- [Pipeline router](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/pipeline_builder.py): `nemo/collections/asr/inference/factory/pipeline_builder.py`
- [Buffered builder](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/buffered_pipeline_builder.py): `nemo/collections/asr/inference/factory/buffered_pipeline_builder.py`
- [Cache-aware builder](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py): `nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py`
- [Buffered pipeline](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py): `nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py`
- [Cache-aware pipeline](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py): `nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py`
- [Pipeline base](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/pipelines/base_pipeline.py): `nemo/collections/asr/inference/pipelines/base_pipeline.py`
- [Stream state](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/streaming/state/state.py): `nemo/collections/asr/inference/streaming/state/state.py`
- [Inference wrappers](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py): `nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py`
- [Text normalization](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/itn/inverse_normalizer.py): `nemo/collections/asr/inference/itn/inverse_normalizer.py`
- [Speech translation](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/inference/nmt/llm_translator.py): `nemo/collections/asr/inference/nmt/llm_translator.py`

### Speech Models

- [ASR models](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/models/asr_model.py): `nemo/collections/asr/models/asr_model.py`
- [Encoders and decoders](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/modules/conformer_encoder.py): `nemo/collections/asr/modules/conformer_encoder.py`
- [ASR objectives](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/losses/ctc.py): `nemo/collections/asr/losses/ctc.py`
- [ASR metrics](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/metrics/wer.py): `nemo/collections/asr/metrics/wer.py`

### Voice Agent

- [WebSocket transport](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py): `nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py`
- [Agent configuration](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/utils/config_manager.py): `nemo/agents/voice_agent/utils/config_manager.py`
- [Streaming recognition](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/services/nemo/streaming_asr.py): `nemo/agents/voice_agent/pipecat/services/nemo/streaming_asr.py`
- [Conversation LLM](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/services/nemo/llm.py): `nemo/agents/voice_agent/pipecat/services/nemo/llm.py`
- [Speech synthesis](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/services/nemo/tts.py): `nemo/agents/voice_agent/pipecat/services/nemo/tts.py`
- [Conversation frames](https://github.com/nvidia-nemo/nemo/blob/main/nemo/agents/voice_agent/pipecat/frames/frames.py): `nemo/agents/voice_agent/pipecat/frames/frames.py`

### Audio Generation

- [Audio transformation](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/audio/models/audio_to_audio.py): `nemo/collections/audio/models/audio_to_audio.py`
- [Audio datasets](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/audio/data/audio_to_audio.py): `nemo/collections/audio/data/audio_to_audio.py`
- [Magpie TTS stages](https://github.com/nvidia-nemo/nemo/blob/main/tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py): `tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py`

### Data and Training

- [Speech datasets](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/asr/data/audio_to_text.py): `nemo/collections/asr/data/audio_to_text.py`
- [Shared data loading](https://github.com/nvidia-nemo/nemo/blob/main/nemo/collections/common/data/lhotse/dataloader.py): `nemo/collections/common/data/lhotse/dataloader.py`
- [Experiment management](https://github.com/nvidia-nemo/nemo/blob/main/nemo/utils/exp_manager.py): `nemo/utils/exp_manager.py`

### Other components

- **Speech client**
- **Transcript**
- **Audio user**

## Connections

- Speech client → Pipeline router: selects pipeline
- Pipeline router → Buffered builder: builds buffered
- Pipeline router → Cache-aware builder: builds cache-aware
- Buffered builder → Buffered pipeline: constructs
- Cache-aware builder → Cache-aware pipeline: constructs
- Buffered pipeline → Pipeline base: uses base
- Cache-aware pipeline → Pipeline base: uses base
- Speech client → Buffered pipeline: streams audio
- Speech client → Cache-aware pipeline: streams audio
- Buffered pipeline → Inference wrappers: runs inference
- Cache-aware pipeline → Inference wrappers: runs inference
- Inference wrappers → ASR models: invokes model
- ASR models → Encoders and decoders: uses components
- Pipeline base → Stream state: tracks streams
- Buffered pipeline → Text normalization: normalizes text
- Buffered pipeline → Speech translation: translates text
- Cache-aware pipeline → Text normalization: normalizes text
- Cache-aware pipeline → Speech translation: translates text
- Buffered pipeline → Transcript: produces transcript
- Cache-aware pipeline → Transcript: produces transcript
- Speech datasets → Shared data loading: uses loading
- Audio datasets → Shared data loading: uses loading
- WebSocket transport → Streaming recognition: dispatches audio
- Agent configuration → Streaming recognition: configures
- Streaming recognition → Conversation frames: emits recognition
- Conversation frames → Conversation LLM: supplies dialogue
- Conversation LLM → Conversation frames: emits response
- Conversation frames → Speech synthesis: supplies text
- Speech synthesis → Audio user: returns speech
- Audio user → Audio transformation: provides audio
- Audio transformation → Audio user: returns audio
- Magpie TTS stages → Audio user: generates speech
- Experiment management → ASR models: supports experiments