How nvidia-nemo/speech works
NeMo Speech is a PyTorch toolkit for building and running speech and audio models, including ASR, TTS, and speech LLMs. A representative user workflow is to configure a streaming ASR request, select a buffered or cache-aware pipeline, process audio into transcript segments, and optionally normalize or translate the text. The tree also shows model/data/training components, audio-to-audio work, and a real-time voice-agent integration. README capabilities guide the high-level branches; only the sampled inference and voice-agent code supports detailed wiring, so other relationships are intentionally limited.
Main entry points: buffered_rnnt_pipeline.py Buffered Streaming · cache_aware_rnnt_pipeline.py Cache-Aware Streaming · pipeline_builder.py Pipeline Router · buffered_pipeline_builder.py Buffered Builder
Components 25
ASR Models
- ASR Models
nemo/collections/asr/models/asr_model.py - Encoders And Decoders
nemo/collections/asr/modules
Inference
- Pipeline Router
nemo/collections/asr/inference/factory/pipeline_builder.py - Buffered Builder
nemo/collections/asr/inference/factory/buffered_pipeline_builder.py - Cache-Aware Builder
nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py - Buffered Streaming
nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py - Cache-Aware Streaming
nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py - Pipeline Base
nemo/collections/asr/inference/pipelines/base_pipeline.py - Stream State
nemo/collections/asr/inference/streaming/state/state.py - ASR Wrappers
nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py - Text Processing
nemo/collections/asr/inference/itn/inverse_normalizer.py - Speech Translation
nemo/collections/asr/inference/nmt/llm_translator.py - Inference Evaluation
nemo/collections/asr/inference/utils/pipeline_eval.py
Data And Training
- Speech Datasets
nemo/collections/asr/data - ASR Losses
nemo/collections/asr/losses - ASR Metrics
nemo/collections/asr/metrics
Audio Models
- Audio-to-Audio Models
nemo/collections/audio/models/audio_to_audio.py - Audio Datasets
nemo/collections/audio/data/audio_to_audio.py - Magpie TTS Stages
tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py
Voice Agent
- WebSocket Transport
nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py - Speech Services
nemo/agents/voice_agent/pipecat/services/nemo - Conversation Frames
nemo/agents/voice_agent/pipecat/frames/frames.py - Agent Configuration
nemo/agents/voice_agent/utils/config_manager.py
Other components
- Researcher / Developer
- Audio Client
Connections 27
- Researcher / Developer → Pipeline Router: configures
- Pipeline Router → Buffered Builder: selects buffered
- Pipeline Router → Cache-Aware Builder: selects cache-aware
- Buffered Builder → Buffered Streaming: constructs
- Cache-Aware Builder → Cache-Aware Streaming: constructs
- Buffered Builder → ASR Wrappers: builds model
- Cache-Aware Builder → ASR Wrappers: builds model
- Buffered Streaming → Pipeline Base: inherits
- Cache-Aware Streaming → Pipeline Base: inherits
- Buffered Streaming → Stream State: updates stream
- Cache-Aware Streaming → Stream State: updates stream
- ASR Wrappers → ASR Models: wraps
- ASR Models → Encoders And Decoders: uses model components
- Researcher / Developer → Speech Datasets: prepares data
- Researcher / Developer → ASR Models: trains or customizes
- Researcher / Developer → Audio Datasets: prepares audio data
- Researcher / Developer → Audio-to-Audio Models: runs audio tasks
- Buffered Streaming → Text Processing: normalizes text
- Buffered Streaming → Speech Translation: translates text
- Cache-Aware Streaming → Text Processing: normalizes text
- Cache-Aware Streaming → Speech Translation: translates text
- Researcher / Developer → Inference Evaluation: evaluates output
- Audio Client → WebSocket Transport: sends audio
- WebSocket Transport → Conversation Frames: dispatches frames
- Agent Configuration → Speech Services: configures services
- Speech Services → Conversation Frames: processes conversation
- Audio Client → Magpie TTS Stages: requests speech