# tsutokiteo/tsukisynth architecture

> Architecture diagram of the GitHub repository tsutokiteo/tsukisynth, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tsutokiteo/tsukisynth
- Repository: https://github.com/tsutokiteo/tsukisynth

## Overview

TsukiSynth is a singing-synthesis engine with WebUI, VST3, and CLI entry points, plus voicebank and singing-style training/tools. The documented synthesis path runs from notes and lyrics through PIT conditioning and sampler/neural pipelines into HiFiSampler rendering, then produces audio and lip-sync data. The graph preserves the documented capability branches, but some internals—especially the neural/refinement stages and plugin bridge—are only described in the README or partial excerpts, so those links are marked unverified.

Main entry points: [`core/engine.py`](https://github.com/tsutokiteo/tsukisynth/blob/main/core/engine.py) (Synthesis engine), [`core/hifisampler/renderer.py`](https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/renderer.py) (HiFiSampler renderer), [`training/core/trainer.py`](https://github.com/tsutokiteo/tsukisynth/blob/main/training/core/trainer.py) (Model trainer), [`core/sampler/voicebank.py`](https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/voicebank.py) (Voicebank samples)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontends["User interfaces"]
  node_webui["WebUI backend<br/>[app.py]"]
  node_api["Engine API<br/>[main.py]"]
  node_plugin["VST3 processor<br/>[processor.h]"]
  node_bridge["Engine bridge<br/>[engine_wrapper.h]"]
  node_cli["Synthesis CLI"]
end

subgraph group_synthesis["Synthesis engine"]
  node_schemas["Synthesis schemas<br/>[schemas.py]"]
  node_engine["Synthesis engine<br/>[engine.py]"]
  node_pit["PIT inference<br/>[onnx_pit.py]"]
  node_sampler["Sampler pipeline<br/>[renderer.py]"]
  node_voicebank[("Voicebank samples<br/>[voicebank.py]")]
  node_world["WORLD rendering<br/>[world_render.py]"]
  node_acoustic["Acoustic model<br/>[model.py]"]
  node_representation["Acoustic representation<br/>[representation.py]"]
  node_hybrid["Pipeline mixing<br/>[hybrid.py]"]
  node_hifi["HiFiSampler renderer<br/>[renderer.py]"]
  node_hificlient["HiFiSampler client<br/>[client.py]"]
  node_hificache[("Feature cache<br/>[cache.py]")]
  node_vocoder["PC HiFi-GAN and HuBERT"]
  node_hifiservice["HiFiSampler service"]
end

subgraph group_training["Training and tools"]
  node_trainingcli["Training CLI<br/>[cli.py]"]
  node_dataset[("Training datasets<br/>[dataset.py]")]
  node_trainconfig["Training config<br/>[config.py]"]
  node_trainer["Model trainer<br/>[trainer.py]"]
  node_progress["Progress tracking<br/>[progress.py]"]
  node_voicebanktool["Voicebank maker<br/>[voicebank_maker.py]"]
  node_pittool["PIT trainer<br/>[pit_trainer.py]"]
  node_converter["Voicebank converter<br/>[converter.py]"]
end

subgraph group_motion["Motion output"]
  node_lip["Lip-sync generator<br/>[lip_sync.py]"]
  node_lipstandalone["Standalone lip sync"]
  node_vrm["VRM motion output<br/>[lip_sync.py]"]
end

node_singer(("Singer"))

node_singer -->|"enters notes"| node_webui
node_singer -->|"runs synthesis"| node_cli
node_singer -->|"plays through"| node_plugin
node_webui -.->|"uses synthesis"| node_engine
node_api -->|"calls engine"| node_engine
node_plugin -->|"uses bridge"| node_bridge
node_bridge -.->|"bridges core"| node_engine
node_cli -.->|"requests synthesis"| node_engine
node_engine -->|"uses types"| node_schemas
node_engine -->|"runs PIT"| node_pit
node_engine -.->|"runs pipeline"| node_sampler
node_engine -->|"loads model"| node_acoustic
node_engine -->|"renders output"| node_hifi
node_engine -->|"generates sync"| node_lip
node_sampler -.->|"uses samples"| node_voicebank
node_sampler -.->|"renders samples"| node_world
node_world -->|"reads samples"| node_voicebank
node_voicebank -->|"provides features"| node_representation
node_hifi -->|"accepts features"| node_representation
node_hifi -->|"mixes pipelines"| node_hybrid
node_hifi -->|"uses cache"| node_hificache
node_hifi -->|"dispatches render"| node_hificlient
node_hificlient -.->|"requests rendering"| node_hifiservice
node_hifi -.->|"refines waveform"| node_vocoder
node_lip -->|"produces frames"| node_vrm
node_webui -->|"generates lip sync"| node_lipstandalone
node_trainingcli -->|"loads config"| node_trainconfig
node_trainingcli -->|"creates dataset"| node_dataset
node_trainingcli -->|"runs training"| node_trainer
node_trainer -->|"trains on data"| node_dataset
node_trainer -->|"uses settings"| node_trainconfig
node_trainer -->|"reports progress"| node_progress
node_singer -->|"creates voicebanks"| node_voicebanktool
node_singer -->|"trains PIT"| node_pittool
node_converter -->|"converts bank"| node_voicebank

click node_webui "https://github.com/tsutokiteo/tsukisynth/blob/main/webui/backend/app.py"
click node_api "https://github.com/tsutokiteo/tsukisynth/blob/main/webui/backend/main.py"
click node_plugin "https://github.com/tsutokiteo/tsukisynth/blob/main/plugin/source/processor.h"
click node_bridge "https://github.com/tsutokiteo/tsukisynth/blob/main/plugin/source/engine_bridge/engine_wrapper.h"
click node_cli "https://github.com/tsutokiteo/tsukisynth/blob/main/singing_engine_cli.py"
click node_schemas "https://github.com/tsutokiteo/tsukisynth/blob/main/core/schemas.py"
click node_engine "https://github.com/tsutokiteo/tsukisynth/blob/main/core/engine.py"
click node_pit "https://github.com/tsutokiteo/tsukisynth/blob/main/core/pit/onnx_pit.py"
click node_sampler "https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/renderer.py"
click node_voicebank "https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/voicebank.py"
click node_world "https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/world_render.py"
click node_acoustic "https://github.com/tsutokiteo/tsukisynth/blob/main/core/acoustic/model.py"
click node_representation "https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/representation.py"
click node_hybrid "https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/hybrid.py"
click node_hifi "https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/renderer.py"
click node_hificlient "https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/client.py"
click node_hificache "https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/cache.py"
click node_lip "https://github.com/tsutokiteo/tsukisynth/blob/main/core/lip/lip_sync.py"
click node_lipstandalone "https://github.com/tsutokiteo/tsukisynth/blob/main/core/lip/lip_sync_standalone.py"
click node_trainingcli "https://github.com/tsutokiteo/tsukisynth/blob/main/training/cli.py"
click node_dataset "https://github.com/tsutokiteo/tsukisynth/blob/main/training/common/dataset.py"
click node_trainconfig "https://github.com/tsutokiteo/tsukisynth/blob/main/training/common/config.py"
click node_trainer "https://github.com/tsutokiteo/tsukisynth/blob/main/training/core/trainer.py"
click node_progress "https://github.com/tsutokiteo/tsukisynth/blob/main/training/common/progress.py"
click node_voicebanktool "https://github.com/tsutokiteo/tsukisynth/blob/main/tools/voicebank_maker.py"
click node_pittool "https://github.com/tsutokiteo/tsukisynth/blob/main/tools/pit_trainer.py"
click node_converter "https://github.com/tsutokiteo/tsukisynth/blob/main/tools/converter.py"
click node_vrm "https://github.com/tsutokiteo/tsukisynth/blob/main/core/lip/lip_sync.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webui,node_api,node_plugin,node_bridge,node_cli toneBlue
class node_schemas,node_engine,node_pit,node_sampler,node_voicebank,node_world,node_acoustic,node_representation,node_hybrid,node_hifi,node_hificlient,node_hificache,node_vocoder,node_hifiservice toneAmber
class node_trainingcli,node_dataset,node_trainconfig,node_trainer,node_progress,node_voicebanktool,node_pittool,node_converter toneMint
class node_lip,node_lipstandalone,node_vrm toneRose
class node_singer toneIndigo
```

## Components

### User interfaces

- [WebUI backend](https://github.com/tsutokiteo/tsukisynth/blob/main/webui/backend/app.py): `webui/backend/app.py`
- [Engine API](https://github.com/tsutokiteo/tsukisynth/blob/main/webui/backend/main.py): `webui/backend/main.py`
- [VST3 processor](https://github.com/tsutokiteo/tsukisynth/blob/main/plugin/source/processor.h): `plugin/source/processor.h`
- [Engine bridge](https://github.com/tsutokiteo/tsukisynth/blob/main/plugin/source/engine_bridge/engine_wrapper.h): `plugin/source/engine_bridge/engine_wrapper.h`
- [Synthesis CLI](https://github.com/tsutokiteo/tsukisynth/blob/main/singing_engine_cli.py): `singing_engine_cli.py`

### Synthesis engine

- [Synthesis schemas](https://github.com/tsutokiteo/tsukisynth/blob/main/core/schemas.py): `core/schemas.py`
- [Synthesis engine](https://github.com/tsutokiteo/tsukisynth/blob/main/core/engine.py): `core/engine.py`
- [PIT inference](https://github.com/tsutokiteo/tsukisynth/blob/main/core/pit/onnx_pit.py): `core/pit/onnx_pit.py`
- [Sampler pipeline](https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/renderer.py): `core/sampler/renderer.py`
- [Voicebank samples](https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/voicebank.py): `core/sampler/voicebank.py`
- [WORLD rendering](https://github.com/tsutokiteo/tsukisynth/blob/main/core/sampler/world_render.py): `core/sampler/world_render.py`
- [Acoustic model](https://github.com/tsutokiteo/tsukisynth/blob/main/core/acoustic/model.py): `core/acoustic/model.py`
- [Acoustic representation](https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/representation.py): `core/hifisampler/representation.py`
- [Pipeline mixing](https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/hybrid.py): `core/hifisampler/hybrid.py`
- [HiFiSampler renderer](https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/renderer.py): `core/hifisampler/renderer.py`
- [HiFiSampler client](https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/client.py): `core/hifisampler/client.py`
- [Feature cache](https://github.com/tsutokiteo/tsukisynth/blob/main/core/hifisampler/cache.py): `core/hifisampler/cache.py`
- **PC HiFi-GAN and HuBERT**
- **HiFiSampler service**

### Training and tools

- [Training CLI](https://github.com/tsutokiteo/tsukisynth/blob/main/training/cli.py): `training/cli.py`
- [Training datasets](https://github.com/tsutokiteo/tsukisynth/blob/main/training/common/dataset.py): `training/common/dataset.py`
- [Training config](https://github.com/tsutokiteo/tsukisynth/blob/main/training/common/config.py): `training/common/config.py`
- [Model trainer](https://github.com/tsutokiteo/tsukisynth/blob/main/training/core/trainer.py): `training/core/trainer.py`
- [Progress tracking](https://github.com/tsutokiteo/tsukisynth/blob/main/training/common/progress.py): `training/common/progress.py`
- [Voicebank maker](https://github.com/tsutokiteo/tsukisynth/blob/main/tools/voicebank_maker.py): `tools/voicebank_maker.py`
- [PIT trainer](https://github.com/tsutokiteo/tsukisynth/blob/main/tools/pit_trainer.py): `tools/pit_trainer.py`
- [Voicebank converter](https://github.com/tsutokiteo/tsukisynth/blob/main/tools/converter.py): `tools/converter.py`

### Motion output

- [Lip-sync generator](https://github.com/tsutokiteo/tsukisynth/blob/main/core/lip/lip_sync.py): `core/lip/lip_sync.py`
- [Standalone lip sync](https://github.com/tsutokiteo/tsukisynth/blob/main/core/lip/lip_sync_standalone.py): `core/lip/lip_sync_standalone.py`
- [VRM motion output](https://github.com/tsutokiteo/tsukisynth/blob/main/core/lip/lip_sync.py): `core/lip/lip_sync.py`

### Other components

- **Singer**

## Connections

- Singer → WebUI backend: enters notes
- Singer → Synthesis CLI: runs synthesis (evidence: `singing_engine_cli.py`)
- Singer → VST3 processor: plays through
- WebUI backend → Synthesis engine: uses synthesis
- Engine API → Synthesis engine: calls engine (evidence: `webui/backend/main.py`)
- VST3 processor → Engine bridge: uses bridge (evidence: `plugin/source/processor.h`)
- Engine bridge → Synthesis engine: bridges core
- Synthesis CLI → Synthesis engine: requests synthesis
- Synthesis engine → Synthesis schemas: uses types (evidence: `core/engine.py`)
- Synthesis engine → PIT inference: runs PIT (evidence: `core/engine.py`)
- Synthesis engine → Sampler pipeline: runs pipeline
- Synthesis engine → Acoustic model: loads model (evidence: `core/engine.py`)
- Synthesis engine → HiFiSampler renderer: renders output (evidence: `core/engine.py`)
- Synthesis engine → Lip-sync generator: generates sync (evidence: `core/engine.py`)
- Sampler pipeline → Voicebank samples: uses samples
- Sampler pipeline → WORLD rendering: renders samples
- WORLD rendering → Voicebank samples: reads samples (evidence: `core/sampler/world_render.py`)
- Voicebank samples → Acoustic representation: provides features (evidence: `core/sampler/voicebank.py`)
- HiFiSampler renderer → Acoustic representation: accepts features (evidence: `core/hifisampler/renderer.py`)
- HiFiSampler renderer → Pipeline mixing: mixes pipelines (evidence: `core/hifisampler/renderer.py`)
- HiFiSampler renderer → Feature cache: uses cache (evidence: `core/hifisampler/renderer.py`)
- HiFiSampler renderer → HiFiSampler client: dispatches render (evidence: `core/hifisampler/renderer.py`)
- HiFiSampler client → HiFiSampler service: requests rendering
- HiFiSampler renderer → PC HiFi-GAN and HuBERT: refines waveform
- Lip-sync generator → VRM motion output: produces frames (evidence: `core/lip/lip_sync.py`)
- WebUI backend → Standalone lip sync: generates lip sync (evidence: `webui/backend/app.py`)
- Training CLI → Training config: loads config (evidence: `training/cli.py`)
- Training CLI → Training datasets: creates dataset (evidence: `training/cli.py`)
- Training CLI → Model trainer: runs training (evidence: `training/cli.py`)
- Model trainer → Training datasets: trains on data (evidence: `training/core/trainer.py`)
- Model trainer → Training config: uses settings (evidence: `training/core/trainer.py`)
- Model trainer → Progress tracking: reports progress (evidence: `training/core/trainer.py`)
- Singer → Voicebank maker: creates voicebanks
- Singer → PIT trainer: trains PIT
- Voicebank converter → Voicebank samples: converts bank (evidence: `tools/converter.py`)
