How k2-fsa/sherpa works
sherpa is a speech-to-text inference and deployment framework. Its main workflow accepts audio, extracts features, runs an offline or streaming recognition model and decoder, and returns transcription results through C++/Python APIs or network serving interfaces. The repository also includes VAD, speaker-embedding inference, and Triton deployment components. The README documents the C++ and Python APIs; sampled source and the SOURCE INDEX support selected implementation relationships. Many runtime files were not sampled, so uncertain links are omitted or shown as unverified.
Main entry points: offline-recognizer.cc Offline recognizer · online-recognizer.cc Online recognizer · offline-stream.h Audio streams · speaker-embedding-extractor.cc Speaker embeddings
Components 22
Recognition runtime
- Offline recognizer
sherpa/cpp_api/offline-recognizer.cc - Online recognizer
sherpa/cpp_api/online-recognizer.cc - Audio streams
sherpa/cpp_api/offline-stream.h - Feature extraction
sherpa/csrc/fbank-features.cc - Offline models
sherpa/csrc/offline-conformer-transducer-model.cc - Streaming models
sherpa/csrc/online-zipformer-transducer-model.cc - Offline decoding
sherpa/csrc/offline-transducer-greedy-search-decoder.cc - Streaming decoding
sherpa/csrc/online-transducer-fast-beam-search-decoder.cc
Speech tasks
- Voice activity
sherpa/csrc/voice-activity-detector.cc - Speaker embeddings
sherpa/csrc/speaker-embedding-extractor.cc - Embedding inference
sherpa/csrc/speaker-embedding-extractor-general-impl.h - Embedding model
sherpa/csrc/speaker-embedding-extractor-model.cc
Client interfaces
- WebSocket clients
sherpa/cpp_api/websocket/online-websocket-client.cc - gRPC client
sherpa/cpp_api/grpc/online-grpc-client.cc
Python API
- Speaker Python bindings
sherpa/python/csrc/speaker-embedding-extractor.cc - Python package
sherpa/python/sherpa/__init__.py - Python HTTP server
sherpa/python/sherpa/http_server.py
Triton serving
- Transducer scorer
triton/transducer-scorer-backend/src/scorer.cc - Triton features
triton/model_repo_streaming/feature_extractor/1/model.py - Whisper Triton
triton/speech_llm/model_repo_whisper_qwen_trtllm/tensorrt_llm/1/model.py
Other components
- Audio user
- Transcript consumer
Connections 26
- Audio user → Offline recognizer: provides audio
- Audio user → Online recognizer: provides audio
- Offline recognizer → Audio streams: creates stream
- Online recognizer → Audio streams: uses stream
- Audio streams → Feature extraction: extracts features
- Offline recognizer → Offline models: runs inference
- Offline recognizer → Offline decoding: decodes output
- Online recognizer → Streaming models: runs inference
- Online recognizer → Streaming decoding: decodes output
- Offline recognizer → Transcript consumer: returns transcript
- Online recognizer → Transcript consumer: returns transcript
- Audio user → Voice activity: submits audio
- Audio user → Speaker embeddings: submits audio
- Speaker embeddings → Embedding inference: delegates
sherpa/csrc/speaker-embedding-extractor.cc - Embedding inference → Embedding model: computes embeddings
sherpa/csrc/speaker-embedding-extractor-general-impl.h - Speaker Python bindings → Speaker embeddings: binds API
sherpa/python/csrc/speaker-embedding-extractor.cc - Audio user → Python package: uses Python API
README.md - Python package → Python HTTP server: imports
sherpa/python/sherpa/__init__.py - Audio user → WebSocket clients: submits audio
sherpa/cpp_api/websocket/online-websocket-client.cc - Audio user → gRPC client: submits audio
sherpa/cpp_api/grpc/online-grpc-client.cc - WebSocket clients → Transcript consumer: receives results
- gRPC client → Transcript consumer: receives results
- Audio user → Triton features: sends audio
- Triton features → Transducer scorer: passes features
- Audio user → Whisper Triton: sends audio
- Whisper Triton → Transcript consumer: returns transcript