How tsutokiteo/tsukisynth works
TsukiSynth is a singing-synthesis engine with WebUI, VST3, and CLI entry points, plus voicebank and singing-style training/tools. The documented synthesis path runs from notes and lyrics through PIT conditioning and sampler/neural pipelines into HiFiSampler rendering, then produces audio and lip-sync data. The graph preserves the documented capability branches, but some internals—especially the neural/refinement stages and plugin bridge—are only described in the README or partial excerpts, so those links are marked unverified.
Main entry points: engine.py Synthesis engine · renderer.py HiFiSampler renderer · trainer.py Model trainer · voicebank.py Voicebank samples
Components 31
User interfaces
- WebUI backend
webui/backend/app.py - Engine API
webui/backend/main.py - VST3 processor
plugin/source/processor.h - Engine bridge
plugin/source/engine_bridge/engine_wrapper.h - Synthesis CLI
singing_engine_cli.py
Synthesis engine
- Synthesis schemas
core/schemas.py - Synthesis engine
core/engine.py - PIT inference
core/pit/onnx_pit.py - Sampler pipeline
core/sampler/renderer.py - Voicebank samples
core/sampler/voicebank.py - WORLD rendering
core/sampler/world_render.py - Acoustic model
core/acoustic/model.py - Acoustic representation
core/hifisampler/representation.py - Pipeline mixing
core/hifisampler/hybrid.py - HiFiSampler renderer
core/hifisampler/renderer.py - HiFiSampler client
core/hifisampler/client.py - Feature cache
core/hifisampler/cache.py - PC HiFi-GAN and HuBERT
- HiFiSampler service
Training and tools
- Training CLI
training/cli.py - Training datasets
training/common/dataset.py - Training config
training/common/config.py - Model trainer
training/core/trainer.py - Progress tracking
training/common/progress.py - Voicebank maker
tools/voicebank_maker.py - PIT trainer
tools/pit_trainer.py - Voicebank converter
tools/converter.py
Motion output
- Lip-sync generator
core/lip/lip_sync.py - Standalone lip sync
core/lip/lip_sync_standalone.py - VRM motion output
core/lip/lip_sync.py
Other components
- Singer
Connections 35
- Singer → WebUI backend: enters notes
- Singer → Synthesis CLI: runs synthesis
singing_engine_cli.py - Singer → VST3 processor: plays through
- WebUI backend → Synthesis engine: uses synthesis
- Engine API → Synthesis engine: calls engine
webui/backend/main.py - VST3 processor → Engine bridge: uses bridge
plugin/source/processor.h - Engine bridge → Synthesis engine: bridges core
- Synthesis CLI → Synthesis engine: requests synthesis
- Synthesis engine → Synthesis schemas: uses types
core/engine.py - Synthesis engine → PIT inference: runs PIT
core/engine.py - Synthesis engine → Sampler pipeline: runs pipeline
- Synthesis engine → Acoustic model: loads model
core/engine.py - Synthesis engine → HiFiSampler renderer: renders output
core/engine.py - Synthesis engine → Lip-sync generator: generates sync
core/engine.py - Sampler pipeline → Voicebank samples: uses samples
- Sampler pipeline → WORLD rendering: renders samples
- WORLD rendering → Voicebank samples: reads samples
core/sampler/world_render.py - Voicebank samples → Acoustic representation: provides features
core/sampler/voicebank.py - HiFiSampler renderer → Acoustic representation: accepts features
core/hifisampler/renderer.py - HiFiSampler renderer → Pipeline mixing: mixes pipelines
core/hifisampler/renderer.py - HiFiSampler renderer → Feature cache: uses cache
core/hifisampler/renderer.py - HiFiSampler renderer → HiFiSampler client: dispatches render
core/hifisampler/renderer.py - HiFiSampler client → HiFiSampler service: requests rendering
- HiFiSampler renderer → PC HiFi-GAN and HuBERT: refines waveform
- Lip-sync generator → VRM motion output: produces frames
core/lip/lip_sync.py - WebUI backend → Standalone lip sync: generates lip sync
webui/backend/app.py - Training CLI → Training config: loads config
training/cli.py - Training CLI → Training datasets: creates dataset
training/cli.py - Training CLI → Model trainer: runs training
training/cli.py - Model trainer → Training datasets: trains on data
training/core/trainer.py - Model trainer → Training config: uses settings
training/core/trainer.py - Model trainer → Progress tracking: reports progress
training/core/trainer.py - Singer → Voicebank maker: creates voicebanks
- Singer → PIT trainer: trains PIT
- Voicebank converter → Voicebank samples: converts bank
tools/converter.py