# k2-fsa/sherpa architecture

> Architecture diagram of the GitHub repository k2-fsa/sherpa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/k2-fsa/sherpa
- Repository: https://github.com/k2-fsa/sherpa

## Overview

sherpa is a speech-to-text inference and deployment framework. Its main workflow accepts audio, extracts features, runs an offline or streaming recognition model and decoder, and returns transcription results through C++/Python APIs or network serving interfaces. The repository also includes VAD, speaker-embedding inference, and Triton deployment components. The README documents the C++ and Python APIs; sampled source and the SOURCE INDEX support selected implementation relationships. Many runtime files were not sampled, so uncertain links are omitted or shown as unverified.

Main entry points: [`sherpa/cpp_api/offline-recognizer.cc`](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/offline-recognizer.cc) (Offline recognizer), [`sherpa/cpp_api/online-recognizer.cc`](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/online-recognizer.cc) (Online recognizer), [`sherpa/cpp_api/offline-stream.h`](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/offline-stream.h) (Audio streams), [`sherpa/csrc/speaker-embedding-extractor.cc`](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor.cc) (Speaker embeddings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_recognition["Recognition runtime"]
  node_offline_api["Offline recognizer"]
  node_online_api["Online recognizer"]
  node_audio_stream["Audio streams<br/>[offline-stream.h]"]
  node_features["Feature extraction<br/>[fbank-features.cc]"]
  node_offline_models["Offline models"]
  node_online_models["Streaming models"]
  node_offline_decode["Offline decoding"]
  node_online_decode["Streaming decoding"]
end

subgraph group_speech_tasks["Speech tasks"]
  node_vad["Voice activity"]
  node_speaker_api["Speaker embeddings"]
  node_speaker_impl["Embedding inference"]
  node_speaker_model["Embedding model"]
end

subgraph group_interfaces["Client interfaces"]
  node_ws_clients["WebSocket clients"]
  node_grpc_client["gRPC client"]
end

subgraph group_python["Python API"]
  node_speaker_bindings["Speaker Python bindings"]
  node_python_package["Python package<br/>[__init__.py]"]
  node_http_server["Python HTTP server<br/>[http_server.py]"]
end

subgraph group_serving["Triton serving"]
  node_triton_scorer["Transducer scorer<br/>[scorer.cc]"]
  node_triton_features["Triton features<br/>[model.py]"]
  node_triton_whisper["Whisper Triton<br/>[model.py]"]
end

node_user(("Audio user"))
node_transcript(("Transcript consumer"))

node_user -.->|"provides audio"| node_offline_api
node_user -.->|"provides audio"| node_online_api
node_offline_api -.->|"creates stream"| node_audio_stream
node_online_api -.->|"uses stream"| node_audio_stream
node_audio_stream -.->|"extracts features"| node_features
node_offline_api -.->|"runs inference"| node_offline_models
node_offline_api -.->|"decodes output"| node_offline_decode
node_online_api -.->|"runs inference"| node_online_models
node_online_api -.->|"decodes output"| node_online_decode
node_offline_api -.->|"returns transcript"| node_transcript
node_online_api -.->|"returns transcript"| node_transcript
node_user -.->|"submits audio"| node_vad
node_user -.->|"submits audio"| node_speaker_api
node_speaker_api -->|"delegates"| node_speaker_impl
node_speaker_impl -->|"computes embeddings"| node_speaker_model
node_speaker_bindings -->|"binds API"| node_speaker_api
node_user -->|"uses Python API"| node_python_package
node_python_package -->|"imports"| node_http_server
node_user -->|"submits audio"| node_ws_clients
node_user -->|"submits audio"| node_grpc_client
node_ws_clients -.->|"receives results"| node_transcript
node_grpc_client -.->|"receives results"| node_transcript
node_user -.->|"sends audio"| node_triton_features
node_triton_features -.->|"passes features"| node_triton_scorer
node_user -.->|"sends audio"| node_triton_whisper
node_triton_whisper -.->|"returns transcript"| node_transcript

click node_offline_api "https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/offline-recognizer.cc"
click node_online_api "https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/online-recognizer.cc"
click node_audio_stream "https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/offline-stream.h"
click node_features "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/fbank-features.cc"
click node_offline_models "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/offline-conformer-transducer-model.cc"
click node_online_models "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/online-zipformer-transducer-model.cc"
click node_offline_decode "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/offline-transducer-greedy-search-decoder.cc"
click node_online_decode "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/online-transducer-fast-beam-search-decoder.cc"
click node_vad "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/voice-activity-detector.cc"
click node_speaker_api "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor.cc"
click node_speaker_impl "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor-general-impl.h"
click node_speaker_model "https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor-model.cc"
click node_speaker_bindings "https://github.com/k2-fsa/sherpa/blob/master/sherpa/python/csrc/speaker-embedding-extractor.cc"
click node_python_package "https://github.com/k2-fsa/sherpa/blob/master/sherpa/python/sherpa/__init__.py"
click node_http_server "https://github.com/k2-fsa/sherpa/blob/master/sherpa/python/sherpa/http_server.py"
click node_ws_clients "https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/websocket/online-websocket-client.cc"
click node_grpc_client "https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/grpc/online-grpc-client.cc"
click node_triton_scorer "https://github.com/k2-fsa/sherpa/blob/master/triton/transducer-scorer-backend/src/scorer.cc"
click node_triton_features "https://github.com/k2-fsa/sherpa/blob/master/triton/model_repo_streaming/feature_extractor/1/model.py"
click node_triton_whisper "https://github.com/k2-fsa/sherpa/blob/master/triton/speech_llm/model_repo_whisper_qwen_trtllm/tensorrt_llm/1/model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_offline_api,node_online_api,node_audio_stream,node_features,node_offline_models,node_online_models,node_offline_decode,node_online_decode,node_user toneBlue
class node_vad,node_speaker_api,node_speaker_impl,node_speaker_model toneAmber
class node_ws_clients,node_grpc_client toneMint
class node_speaker_bindings,node_python_package,node_http_server toneRose
class node_triton_scorer,node_triton_features,node_triton_whisper,node_transcript toneIndigo
```

## Components

### Recognition runtime

- [Offline recognizer](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/offline-recognizer.cc): `sherpa/cpp_api/offline-recognizer.cc`
- [Online recognizer](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/online-recognizer.cc): `sherpa/cpp_api/online-recognizer.cc`
- [Audio streams](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/offline-stream.h): `sherpa/cpp_api/offline-stream.h`
- [Feature extraction](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/fbank-features.cc): `sherpa/csrc/fbank-features.cc`
- [Offline models](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/offline-conformer-transducer-model.cc): `sherpa/csrc/offline-conformer-transducer-model.cc`
- [Streaming models](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/online-zipformer-transducer-model.cc): `sherpa/csrc/online-zipformer-transducer-model.cc`
- [Offline decoding](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/offline-transducer-greedy-search-decoder.cc): `sherpa/csrc/offline-transducer-greedy-search-decoder.cc`
- [Streaming decoding](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/online-transducer-fast-beam-search-decoder.cc): `sherpa/csrc/online-transducer-fast-beam-search-decoder.cc`

### Speech tasks

- [Voice activity](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/voice-activity-detector.cc): `sherpa/csrc/voice-activity-detector.cc`
- [Speaker embeddings](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor.cc): `sherpa/csrc/speaker-embedding-extractor.cc`
- [Embedding inference](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor-general-impl.h): `sherpa/csrc/speaker-embedding-extractor-general-impl.h`
- [Embedding model](https://github.com/k2-fsa/sherpa/blob/master/sherpa/csrc/speaker-embedding-extractor-model.cc): `sherpa/csrc/speaker-embedding-extractor-model.cc`

### Client interfaces

- [WebSocket clients](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/websocket/online-websocket-client.cc): `sherpa/cpp_api/websocket/online-websocket-client.cc`
- [gRPC client](https://github.com/k2-fsa/sherpa/blob/master/sherpa/cpp_api/grpc/online-grpc-client.cc): `sherpa/cpp_api/grpc/online-grpc-client.cc`

### Python API

- [Speaker Python bindings](https://github.com/k2-fsa/sherpa/blob/master/sherpa/python/csrc/speaker-embedding-extractor.cc): `sherpa/python/csrc/speaker-embedding-extractor.cc`
- [Python package](https://github.com/k2-fsa/sherpa/blob/master/sherpa/python/sherpa/__init__.py): `sherpa/python/sherpa/__init__.py`
- [Python HTTP server](https://github.com/k2-fsa/sherpa/blob/master/sherpa/python/sherpa/http_server.py): `sherpa/python/sherpa/http_server.py`

### Triton serving

- [Transducer scorer](https://github.com/k2-fsa/sherpa/blob/master/triton/transducer-scorer-backend/src/scorer.cc): `triton/transducer-scorer-backend/src/scorer.cc`
- [Triton features](https://github.com/k2-fsa/sherpa/blob/master/triton/model_repo_streaming/feature_extractor/1/model.py): `triton/model_repo_streaming/feature_extractor/1/model.py`
- [Whisper Triton](https://github.com/k2-fsa/sherpa/blob/master/triton/speech_llm/model_repo_whisper_qwen_trtllm/tensorrt_llm/1/model.py): `triton/speech_llm/model_repo_whisper_qwen_trtllm/tensorrt_llm/1/model.py`

### Other components

- **Audio user**
- **Transcript consumer**

## Connections

- Audio user → Offline recognizer: provides audio
- Audio user → Online recognizer: provides audio
- Offline recognizer → Audio streams: creates stream
- Online recognizer → Audio streams: uses stream
- Audio streams → Feature extraction: extracts features
- Offline recognizer → Offline models: runs inference
- Offline recognizer → Offline decoding: decodes output
- Online recognizer → Streaming models: runs inference
- Online recognizer → Streaming decoding: decodes output
- Offline recognizer → Transcript consumer: returns transcript
- Online recognizer → Transcript consumer: returns transcript
- Audio user → Voice activity: submits audio
- Audio user → Speaker embeddings: submits audio
- Speaker embeddings → Embedding inference: delegates (evidence: `sherpa/csrc/speaker-embedding-extractor.cc`)
- Embedding inference → Embedding model: computes embeddings (evidence: `sherpa/csrc/speaker-embedding-extractor-general-impl.h`)
- Speaker Python bindings → Speaker embeddings: binds API (evidence: `sherpa/python/csrc/speaker-embedding-extractor.cc`)
- Audio user → Python package: uses Python API (evidence: `README.md`)
- Python package → Python HTTP server: imports (evidence: `sherpa/python/sherpa/__init__.py`)
- Audio user → WebSocket clients: submits audio (evidence: `sherpa/cpp_api/websocket/online-websocket-client.cc`)
- Audio user → gRPC client: submits audio (evidence: `sherpa/cpp_api/grpc/online-grpc-client.cc`)
- WebSocket clients → Transcript consumer: receives results
- gRPC client → Transcript consumer: receives results
- Audio user → Triton features: sends audio
- Triton features → Transducer scorer: passes features
- Audio user → Whisper Triton: sends audio
- Whisper Triton → Transcript consumer: returns transcript
