# nvidia-nemo/speech architecture

> Architecture diagram of the GitHub repository nvidia-nemo/speech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia-nemo/speech
- Repository: https://github.com/nvidia-nemo/speech

## Overview

NeMo Speech is a PyTorch toolkit for building and running speech and audio models, including ASR, TTS, and speech LLMs. A representative user workflow is to configure a streaming ASR request, select a buffered or cache-aware pipeline, process audio into transcript segments, and optionally normalize or translate the text. The tree also shows model/data/training components, audio-to-audio work, and a real-time voice-agent integration. README capabilities guide the high-level branches; only the sampled inference and voice-agent code supports detailed wiring, so other relationships are intentionally limited.

Main entry points: [`nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py`](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py) (Buffered Streaming), [`nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py`](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py) (Cache-Aware Streaming), [`nemo/collections/asr/inference/factory/pipeline_builder.py`](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/pipeline_builder.py) (Pipeline Router), [`nemo/collections/asr/inference/factory/buffered_pipeline_builder.py`](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/buffered_pipeline_builder.py) (Buffered Builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_asr["ASR Models"]
  node_asr_models["ASR Models<br/>[asr_model.py]"]
  node_asr_modules["Encoders And Decoders"]
end

subgraph group_infer["Inference"]
  node_pipeline_builder["Pipeline Router"]
  node_buffered_builder["Buffered Builder"]
  node_cache_builder["Cache-Aware Builder"]
  node_buffered_pipeline["Buffered Streaming"]
  node_cache_pipeline["Cache-Aware Streaming"]
  node_base_pipeline["Pipeline Base<br/>[base_pipeline.py]"]
  node_stream_state["Stream State<br/>[state.py]"]
  node_inference_wrappers["ASR Wrappers"]
  node_postprocess["Text Processing"]
  node_translation["Speech Translation<br/>[llm_translator.py]"]
  node_eval["Inference Evaluation<br/>[pipeline_eval.py]"]
end

subgraph group_training["Data And Training"]
  node_asr_data["Speech Datasets"]
  node_asr_losses["ASR Losses"]
  node_asr_metrics["ASR Metrics"]
end

subgraph group_audio["Audio Models"]
  node_audio_models["Audio-to-Audio Models<br/>[audio_to_audio.py]"]
  node_audio_data["Audio Datasets<br/>[audio_to_audio.py]"]
  node_magpie_pipeline["Magpie TTS Stages<br/>[pipeline.py]"]
end

subgraph group_realtime["Voice Agent"]
  node_voice_transport["WebSocket Transport"]
  node_voice_services["Speech Services"]
  node_voice_frames["Conversation Frames<br/>[frames.py]"]
  node_voice_config["Agent Configuration<br/>[config_manager.py]"]
end

node_user(("Researcher / Developer"))
node_client(("Audio Client"))

node_user -->|"configures"| node_pipeline_builder
node_pipeline_builder -->|"selects buffered"| node_buffered_builder
node_pipeline_builder -->|"selects cache-aware"| node_cache_builder
node_buffered_builder -->|"constructs"| node_buffered_pipeline
node_cache_builder -->|"constructs"| node_cache_pipeline
node_buffered_builder -->|"builds model"| node_inference_wrappers
node_cache_builder -->|"builds model"| node_inference_wrappers
node_buffered_pipeline -->|"inherits"| node_base_pipeline
node_cache_pipeline -->|"inherits"| node_base_pipeline
node_buffered_pipeline -->|"updates stream"| node_stream_state
node_cache_pipeline -->|"updates stream"| node_stream_state
node_inference_wrappers -->|"wraps"| node_asr_models
node_asr_models -->|"uses model components"| node_asr_modules
node_user -->|"prepares data"| node_asr_data
node_user -->|"trains or customizes"| node_asr_models
node_user -->|"prepares audio data"| node_audio_data
node_user -->|"runs audio tasks"| node_audio_models
node_buffered_pipeline -->|"normalizes text"| node_postprocess
node_buffered_pipeline -->|"translates text"| node_translation
node_cache_pipeline -->|"normalizes text"| node_postprocess
node_cache_pipeline -->|"translates text"| node_translation
node_user -->|"evaluates output"| node_eval
node_client -->|"sends audio"| node_voice_transport
node_voice_transport -->|"dispatches frames"| node_voice_frames
node_voice_config -->|"configures services"| node_voice_services
node_voice_services -->|"processes conversation"| node_voice_frames
node_client -->|"requests speech"| node_magpie_pipeline

click node_pipeline_builder "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/pipeline_builder.py"
click node_buffered_builder "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/buffered_pipeline_builder.py"
click node_cache_builder "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py"
click node_buffered_pipeline "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py"
click node_cache_pipeline "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py"
click node_base_pipeline "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/base_pipeline.py"
click node_stream_state "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/streaming/state/state.py"
click node_inference_wrappers "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py"
click node_asr_models "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/models/asr_model.py"
click node_asr_modules "https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/modules"
click node_asr_data "https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/data"
click node_asr_losses "https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/losses"
click node_asr_metrics "https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/metrics"
click node_postprocess "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/itn/inverse_normalizer.py"
click node_translation "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/nmt/llm_translator.py"
click node_eval "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/utils/pipeline_eval.py"
click node_audio_models "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/audio/models/audio_to_audio.py"
click node_audio_data "https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/audio/data/audio_to_audio.py"
click node_voice_transport "https://github.com/nvidia-nemo/speech/blob/main/nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py"
click node_voice_services "https://github.com/nvidia-nemo/speech/tree/main/nemo/agents/voice_agent/pipecat/services/nemo"
click node_voice_frames "https://github.com/nvidia-nemo/speech/blob/main/nemo/agents/voice_agent/pipecat/frames/frames.py"
click node_voice_config "https://github.com/nvidia-nemo/speech/blob/main/nemo/agents/voice_agent/utils/config_manager.py"
click node_magpie_pipeline "https://github.com/nvidia-nemo/speech/blob/main/tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_asr_models,node_asr_modules,node_client toneBlue
class node_pipeline_builder,node_buffered_builder,node_cache_builder,node_buffered_pipeline,node_cache_pipeline,node_base_pipeline,node_stream_state,node_inference_wrappers,node_postprocess,node_translation,node_eval toneAmber
class node_asr_data,node_asr_losses,node_asr_metrics toneMint
class node_audio_models,node_audio_data,node_magpie_pipeline toneRose
class node_voice_transport,node_voice_services,node_voice_frames,node_voice_config,node_user toneIndigo
```

## Components

### ASR Models

- [ASR Models](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/models/asr_model.py): `nemo/collections/asr/models/asr_model.py`
- [Encoders And Decoders](https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/modules): `nemo/collections/asr/modules`

### Inference

- [Pipeline Router](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/pipeline_builder.py): `nemo/collections/asr/inference/factory/pipeline_builder.py`
- [Buffered Builder](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/buffered_pipeline_builder.py): `nemo/collections/asr/inference/factory/buffered_pipeline_builder.py`
- [Cache-Aware Builder](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py): `nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py`
- [Buffered Streaming](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py): `nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py`
- [Cache-Aware Streaming](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py): `nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py`
- [Pipeline Base](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/pipelines/base_pipeline.py): `nemo/collections/asr/inference/pipelines/base_pipeline.py`
- [Stream State](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/streaming/state/state.py): `nemo/collections/asr/inference/streaming/state/state.py`
- [ASR Wrappers](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py): `nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py`
- [Text Processing](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/itn/inverse_normalizer.py): `nemo/collections/asr/inference/itn/inverse_normalizer.py`
- [Speech Translation](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/nmt/llm_translator.py): `nemo/collections/asr/inference/nmt/llm_translator.py`
- [Inference Evaluation](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/asr/inference/utils/pipeline_eval.py): `nemo/collections/asr/inference/utils/pipeline_eval.py`

### Data And Training

- [Speech Datasets](https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/data): `nemo/collections/asr/data`
- [ASR Losses](https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/losses): `nemo/collections/asr/losses`
- [ASR Metrics](https://github.com/nvidia-nemo/speech/tree/main/nemo/collections/asr/metrics): `nemo/collections/asr/metrics`

### Audio Models

- [Audio-to-Audio Models](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/audio/models/audio_to_audio.py): `nemo/collections/audio/models/audio_to_audio.py`
- [Audio Datasets](https://github.com/nvidia-nemo/speech/blob/main/nemo/collections/audio/data/audio_to_audio.py): `nemo/collections/audio/data/audio_to_audio.py`
- [Magpie TTS Stages](https://github.com/nvidia-nemo/speech/blob/main/tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py): `tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py`

### Voice Agent

- [WebSocket Transport](https://github.com/nvidia-nemo/speech/blob/main/nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py): `nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py`
- [Speech Services](https://github.com/nvidia-nemo/speech/tree/main/nemo/agents/voice_agent/pipecat/services/nemo): `nemo/agents/voice_agent/pipecat/services/nemo`
- [Conversation Frames](https://github.com/nvidia-nemo/speech/blob/main/nemo/agents/voice_agent/pipecat/frames/frames.py): `nemo/agents/voice_agent/pipecat/frames/frames.py`
- [Agent Configuration](https://github.com/nvidia-nemo/speech/blob/main/nemo/agents/voice_agent/utils/config_manager.py): `nemo/agents/voice_agent/utils/config_manager.py`

### Other components

- **Researcher / Developer**
- **Audio Client**

## Connections

- Researcher / Developer → Pipeline Router: configures
- Pipeline Router → Buffered Builder: selects buffered
- Pipeline Router → Cache-Aware Builder: selects cache-aware
- Buffered Builder → Buffered Streaming: constructs
- Cache-Aware Builder → Cache-Aware Streaming: constructs
- Buffered Builder → ASR Wrappers: builds model
- Cache-Aware Builder → ASR Wrappers: builds model
- Buffered Streaming → Pipeline Base: inherits
- Cache-Aware Streaming → Pipeline Base: inherits
- Buffered Streaming → Stream State: updates stream
- Cache-Aware Streaming → Stream State: updates stream
- ASR Wrappers → ASR Models: wraps
- ASR Models → Encoders And Decoders: uses model components
- Researcher / Developer → Speech Datasets: prepares data
- Researcher / Developer → ASR Models: trains or customizes
- Researcher / Developer → Audio Datasets: prepares audio data
- Researcher / Developer → Audio-to-Audio Models: runs audio tasks
- Buffered Streaming → Text Processing: normalizes text
- Buffered Streaming → Speech Translation: translates text
- Cache-Aware Streaming → Text Processing: normalizes text
- Cache-Aware Streaming → Speech Translation: translates text
- Researcher / Developer → Inference Evaluation: evaluates output
- Audio Client → WebSocket Transport: sends audio
- WebSocket Transport → Conversation Frames: dispatches frames
- Agent Configuration → Speech Services: configures services
- Speech Services → Conversation Frames: processes conversation
- Audio Client → Magpie TTS Stages: requests speech