# pengchengguo/espnet architecture

> Architecture diagram of the GitHub repository pengchengguo/espnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pengchengguo/espnet
- Repository: https://github.com/pengchengguo/espnet

## Overview

ESPnet is a speech-processing toolkit whose user-facing result is a trained model or an inference output: recognized text, synthesized speech, translated text, enhanced/separated audio, or converted voice. Users prepare recipe data, train a task model, then decode or synthesize. The repository supports both ESPnet1 and ESPnet2; the graph emphasizes ESPnet2’s task-based path and preserves ESPnet1’s major task branches. Relationships involving unsampled subsystems are included only where the README or SOURCE INDEX supports them.

Main entry points: [`espnet2/bin/asr_inference.py`](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/asr_inference.py) (ASR decoding), [`espnet2/bin/tts_inference.py`](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/tts_inference.py) (TTS decoding), [`espnet2/bin/enh_inference.py`](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/enh_inference.py) (Enhancement decoding), [`espnet2/bin/asr_train.py`](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/asr_train.py) (ESPnet2 training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User workflows"]
  node_results["Task outputs"]
  node_audio["Input audio and text"]
end

subgraph group_data["Data and training"]
  node_recipes["Recipe data prep"]
  node_inputdata["Kaldi-style data"]
  node_taskconfig["Task configuration"]
  node_train["ESPnet2 training<br/>[asr_train.py]"]
  node_trainer["Training engine<br/>[trainer.py]"]
  node_espnetmodel["Task model base"]
end

subgraph group_speech["Speech tasks"]
  node_asr["Speech recognition<br/>[espnet_model.py]"]
  node_enhancement["Enhancement and separation<br/>[espnet_model.py]"]
  node_tts["Speech synthesis<br/>[transformer.py]"]
  node_vc["Voice conversion<br/>[vc.py]"]
end

subgraph group_texttasks["Text tasks"]
  node_st["Speech translation<br/>[st.py]"]
  node_mt["Machine translation<br/>[mt.py]"]
  node_lm["Language modeling<br/>[espnet_model.py]"]
end

subgraph group_shared["Shared inference"]
  node_asrinfer["ASR decoding<br/>[asr_inference.py]"]
  node_ttsinfer["TTS decoding<br/>[tts_inference.py]"]
  node_enhanceinfer["Enhancement decoding<br/>[enh_inference.py]"]
  node_tokenizer["Text tokenization<br/>[build_tokenizer.py]"]
  node_beam["Beam search<br/>[beam_search.py]"]
  node_features["Feature and audio I/O<br/>[io_utils.py]"]
end

node_user(("Researcher"))

node_user -->|"prepares data"| node_recipes
node_recipes -->|"produces data"| node_inputdata
node_user -->|"chooses task"| node_taskconfig
node_inputdata -.->|"supplies examples"| node_train
node_taskconfig -->|"configures training"| node_train
node_train -.->|"runs training"| node_trainer
node_trainer -->|"calls model"| node_espnetmodel
node_espnetmodel -->|"task implementation"| node_asr
node_user -->|"provides input"| node_audio
node_audio -->|"decodes speech"| node_asrinfer
node_asrinfer -->|"uses search"| node_beam
node_asrinfer -->|"converts tokens"| node_tokenizer
node_asrinfer -->|"writes recognition"| node_results
node_audio -->|"provides text"| node_ttsinfer
node_ttsinfer -->|"builds TTS model"| node_tts
node_ttsinfer -->|"returns speech"| node_results
node_audio -->|"provides audio"| node_enhanceinfer
node_enhanceinfer -->|"loads task model"| node_enhancement
node_enhanceinfer -.->|"writes enhanced audio"| node_results
node_st -->|"produces translation"| node_results
node_mt -->|"produces translation"| node_results
node_vc -->|"produces converted voice"| node_results
node_lm -.->|"scores hypotheses"| node_asrinfer
node_features -->|"loads speech data"| node_audio

click node_recipes "https://github.com/pengchengguo/espnet/tree/master/egs"
click node_taskconfig "https://github.com/pengchengguo/espnet/tree/master/espnet2/tasks"
click node_train "https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/asr_train.py"
click node_trainer "https://github.com/pengchengguo/espnet/blob/master/espnet2/train/trainer.py"
click node_espnetmodel "https://github.com/pengchengguo/espnet/blob/master/espnet2/train/abs_espnet_model.py"
click node_asr "https://github.com/pengchengguo/espnet/blob/master/espnet2/asr/espnet_model.py"
click node_enhancement "https://github.com/pengchengguo/espnet/blob/master/espnet2/enh/espnet_model.py"
click node_tts "https://github.com/pengchengguo/espnet/blob/master/espnet2/tts/transformer.py"
click node_vc "https://github.com/pengchengguo/espnet/blob/master/espnet/vc/pytorch_backend/vc.py"
click node_st "https://github.com/pengchengguo/espnet/blob/master/espnet/st/pytorch_backend/st.py"
click node_mt "https://github.com/pengchengguo/espnet/blob/master/espnet/mt/pytorch_backend/mt.py"
click node_lm "https://github.com/pengchengguo/espnet/blob/master/espnet2/lm/espnet_model.py"
click node_asrinfer "https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/asr_inference.py"
click node_ttsinfer "https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/tts_inference.py"
click node_enhanceinfer "https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/enh_inference.py"
click node_tokenizer "https://github.com/pengchengguo/espnet/blob/master/espnet2/text/build_tokenizer.py"
click node_beam "https://github.com/pengchengguo/espnet/blob/master/espnet/nets/beam_search.py"
click node_features "https://github.com/pengchengguo/espnet/blob/master/espnet/utils/io_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_results,node_audio toneBlue
class node_recipes,node_inputdata,node_taskconfig,node_train,node_trainer,node_espnetmodel toneAmber
class node_asr,node_enhancement,node_tts,node_vc toneMint
class node_st,node_mt,node_lm toneRose
class node_asrinfer,node_ttsinfer,node_enhanceinfer,node_tokenizer,node_beam,node_features,node_user toneIndigo
```

## Components

### User workflows

- **Task outputs**
- **Input audio and text**

### Data and training

- [Recipe data prep](https://github.com/pengchengguo/espnet/tree/master/egs): `egs`
- **Kaldi-style data**
- [Task configuration](https://github.com/pengchengguo/espnet/tree/master/espnet2/tasks): `espnet2/tasks`
- [ESPnet2 training](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/asr_train.py): `espnet2/bin/asr_train.py`
- [Training engine](https://github.com/pengchengguo/espnet/blob/master/espnet2/train/trainer.py): `espnet2/train/trainer.py`
- [Task model base](https://github.com/pengchengguo/espnet/blob/master/espnet2/train/abs_espnet_model.py): `espnet2/train/abs_espnet_model.py`

### Speech tasks

- [Speech recognition](https://github.com/pengchengguo/espnet/blob/master/espnet2/asr/espnet_model.py): `espnet2/asr/espnet_model.py`
- [Enhancement and separation](https://github.com/pengchengguo/espnet/blob/master/espnet2/enh/espnet_model.py): `espnet2/enh/espnet_model.py`
- [Speech synthesis](https://github.com/pengchengguo/espnet/blob/master/espnet2/tts/transformer.py): `espnet2/tts/transformer.py`
- [Voice conversion](https://github.com/pengchengguo/espnet/blob/master/espnet/vc/pytorch_backend/vc.py): `espnet/vc/pytorch_backend/vc.py`

### Text tasks

- [Speech translation](https://github.com/pengchengguo/espnet/blob/master/espnet/st/pytorch_backend/st.py): `espnet/st/pytorch_backend/st.py`
- [Machine translation](https://github.com/pengchengguo/espnet/blob/master/espnet/mt/pytorch_backend/mt.py): `espnet/mt/pytorch_backend/mt.py`
- [Language modeling](https://github.com/pengchengguo/espnet/blob/master/espnet2/lm/espnet_model.py): `espnet2/lm/espnet_model.py`

### Shared inference

- [ASR decoding](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/asr_inference.py): `espnet2/bin/asr_inference.py`
- [TTS decoding](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/tts_inference.py): `espnet2/bin/tts_inference.py`
- [Enhancement decoding](https://github.com/pengchengguo/espnet/blob/master/espnet2/bin/enh_inference.py): `espnet2/bin/enh_inference.py`
- [Text tokenization](https://github.com/pengchengguo/espnet/blob/master/espnet2/text/build_tokenizer.py): `espnet2/text/build_tokenizer.py`
- [Beam search](https://github.com/pengchengguo/espnet/blob/master/espnet/nets/beam_search.py): `espnet/nets/beam_search.py`
- [Feature and audio I/O](https://github.com/pengchengguo/espnet/blob/master/espnet/utils/io_utils.py): `espnet/utils/io_utils.py`

### Other components

- **Researcher**

## Connections

- Researcher → Recipe data prep: prepares data
- Recipe data prep → Kaldi-style data: produces data
- Researcher → Task configuration: chooses task
- Kaldi-style data → ESPnet2 training: supplies examples
- Task configuration → ESPnet2 training: configures training (evidence: `espnet2/tasks/abs_task.py`)
- ESPnet2 training → Training engine: runs training
- Training engine → Task model base: calls model (evidence: `espnet2/train/trainer.py`)
- Task model base → Speech recognition: task implementation (evidence: `espnet2/asr/espnet_model.py`)
- Researcher → Input audio and text: provides input
- Input audio and text → ASR decoding: decodes speech (evidence: `espnet2/bin/asr_inference.py`)
- ASR decoding → Beam search: uses search (evidence: `espnet2/bin/asr_inference.py`)
- ASR decoding → Text tokenization: converts tokens (evidence: `espnet2/bin/asr_inference.py`)
- ASR decoding → Task outputs: writes recognition (evidence: `espnet2/bin/asr_inference.py`)
- Input audio and text → TTS decoding: provides text
- TTS decoding → Speech synthesis: builds TTS model (evidence: `espnet2/bin/tts_inference.py`)
- TTS decoding → Task outputs: returns speech (evidence: `espnet2/bin/tts_inference.py`)
- Input audio and text → Enhancement decoding: provides audio (evidence: `espnet2/bin/enh_inference.py`)
- Enhancement decoding → Enhancement and separation: loads task model (evidence: `espnet2/bin/enh_inference.py`)
- Enhancement decoding → Task outputs: writes enhanced audio
- Speech translation → Task outputs: produces translation
- Machine translation → Task outputs: produces translation
- Voice conversion → Task outputs: produces converted voice
- Language modeling → ASR decoding: scores hypotheses
- Feature and audio I/O → Input audio and text: loads speech data (evidence: `espnet/utils/io_utils.py`)
