How nvidia-nemo/nemo works
NeMo Speech is a PyTorch toolkit for building, adapting, and deploying speech and audio models. The principal workflow shown here is configured streaming ASR: a caller selects a pipeline, the builder constructs a buffered or cache-aware pipeline, and streaming requests are processed into transcripts with optional inverse text normalization and translation. The repository also contains training/data foundations, audio-generation and enhancement capabilities, and a real-time voice-agent integration. The README establishes these broader capabilities; sampled source excerpts are concentrated on ASR inference, so uncertain wiring among other subsystems is intentionally omitted.
Main entry points: buffered_rnnt_pipeline.py Buffered pipeline · cache_aware_rnnt_pipeline.py Cache-aware pipeline · frames.py Conversation frames · pipeline_builder.py Pipeline router
Components 29
Streaming ASR
- Pipeline router
nemo/collections/asr/inference/factory/pipeline_builder.py - Buffered builder
nemo/collections/asr/inference/factory/buffered_pipeline_builder.py - Cache-aware builder
nemo/collections/asr/inference/factory/cache_aware_pipeline_builder.py - Buffered pipeline
nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py - Cache-aware pipeline
nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py - Pipeline base
nemo/collections/asr/inference/pipelines/base_pipeline.py - Stream state
nemo/collections/asr/inference/streaming/state/state.py - Inference wrappers
nemo/collections/asr/inference/model_wrappers/asr_inference_wrapper.py - Text normalization
nemo/collections/asr/inference/itn/inverse_normalizer.py - Speech translation
nemo/collections/asr/inference/nmt/llm_translator.py
Speech Models
- ASR models
nemo/collections/asr/models/asr_model.py - Encoders and decoders
nemo/collections/asr/modules/conformer_encoder.py - ASR objectives
nemo/collections/asr/losses/ctc.py - ASR metrics
nemo/collections/asr/metrics/wer.py
Voice Agent
- WebSocket transport
nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py - Agent configuration
nemo/agents/voice_agent/utils/config_manager.py - Streaming recognition
nemo/agents/voice_agent/pipecat/services/nemo/streaming_asr.py - Conversation LLM
nemo/agents/voice_agent/pipecat/services/nemo/llm.py - Speech synthesis
nemo/agents/voice_agent/pipecat/services/nemo/tts.py - Conversation frames
nemo/agents/voice_agent/pipecat/frames/frames.py
Audio Generation
- Audio transformation
nemo/collections/audio/models/audio_to_audio.py - Audio datasets
nemo/collections/audio/data/audio_to_audio.py - Magpie TTS stages
tools/easymagpie_vllm_omni/easymagpie_vllm_omni/pipeline.py
Data and Training
- Speech datasets
nemo/collections/asr/data/audio_to_text.py - Shared data loading
nemo/collections/common/data/lhotse/dataloader.py - Experiment management
nemo/utils/exp_manager.py
Other components
- Speech client
- Transcript
- Audio user
Connections 33
- Speech client → Pipeline router: selects pipeline
- Pipeline router → Buffered builder: builds buffered
- Pipeline router → Cache-aware builder: builds cache-aware
- Buffered builder → Buffered pipeline: constructs
- Cache-aware builder → Cache-aware pipeline: constructs
- Buffered pipeline → Pipeline base: uses base
- Cache-aware pipeline → Pipeline base: uses base
- Speech client → Buffered pipeline: streams audio
- Speech client → Cache-aware pipeline: streams audio
- Buffered pipeline → Inference wrappers: runs inference
- Cache-aware pipeline → Inference wrappers: runs inference
- Inference wrappers → ASR models: invokes model
- ASR models → Encoders and decoders: uses components
- Pipeline base → Stream state: tracks streams
- Buffered pipeline → Text normalization: normalizes text
- Buffered pipeline → Speech translation: translates text
- Cache-aware pipeline → Text normalization: normalizes text
- Cache-aware pipeline → Speech translation: translates text
- Buffered pipeline → Transcript: produces transcript
- Cache-aware pipeline → Transcript: produces transcript
- Speech datasets → Shared data loading: uses loading
- Audio datasets → Shared data loading: uses loading
- WebSocket transport → Streaming recognition: dispatches audio
- Agent configuration → Streaming recognition: configures
- Streaming recognition → Conversation frames: emits recognition
- Conversation frames → Conversation LLM: supplies dialogue
- Conversation LLM → Conversation frames: emits response
- Conversation frames → Speech synthesis: supplies text
- Speech synthesis → Audio user: returns speech
- Audio user → Audio transformation: provides audio
- Audio transformation → Audio user: returns audio
- Magpie TTS stages → Audio user: generates speech
- Experiment management → ASR models: supports experiments