How fishaudio/fish-speech works
Fish Speech is a text-to-speech system for expressive, multilingual speech generation and voice cloning, with S2-Pro and Drama 3 model families. The principal workflow accepts text and optional reference audio through an API, prepares prompts, generates semantic/audio codes, decodes them to audio, and streams the result. The repository also includes a WebUI and model capabilities documented in the README, including multi-speaker and long-form generation; editing and multi-turn revision are documented for Drama 3. Several runtime connections are grounded in imports or excerpts; unsampled UI wiring is not asserted.
Main entry points: __init__.py Inference Engine · views.py Request Views · inference.py Semantic Inference · api_utils.py API Utilities
Components 19
User Interfaces
- WebUI
awesome_webui/src/App.tsx - API Client
tools/api_client.py
API Serving
- API Server
tools/api_server.py - Request Views
tools/server/views.py - API Utilities
tools/server/api_utils.py - Request Schema
fish_speech/utils/schema.py - Model Manager
tools/server/model_manager.py - Inference Wrapper
tools/server/inference.py
Speech Generation
- Inference Engine
fish_speech/inference_engine/__init__.py - Semantic Inference
fish_speech/models/text2semantic/inference.py - Drama Generation
fish_speech/models/text2semantic/fish_omni/generate.py - Audio Decoder Tower
fish_speech/models/text2semantic/fish_omni/tower.py - Content Sequences
fish_speech/content_sequence.py - Fish Tokenizer
fish_speech/tokenizer.py
Audio and Prompts
- Reference Loader
fish_speech/inference_engine/reference_loader.py - VQ Manager
fish_speech/inference_engine/vq_manager.py - DAC Codec
fish_speech/models/dac/modded_dac.py - Audio Files
Other components
- User
Connections 22
- User → API Client: submits text
- User → WebUI: uses
- API Client → API Server: sends requests
- API Server → Request Views: routes to
tools/api_server.py - Request Views → API Utilities: calls
tools/server/views.py - Request Views → Model Manager: uses
tools/server/views.py - Request Views → Inference Wrapper: calls
tools/server/views.py - API Utilities → Inference Engine: uses
tools/server/api_utils.py - API Utilities → Request Schema: validates
tools/server/api_utils.py - Model Manager → Inference Engine: initializes
tools/server/model_manager.py - Model Manager → Semantic Inference: loads model
tools/server/model_manager.py - Inference Wrapper → Inference Engine: consumes results
tools/server/inference.py - Inference Engine → Reference Loader: loads prompts
fish_speech/inference_engine/__init__.py - Inference Engine → VQ Manager: decodes codes
fish_speech/inference_engine/__init__.py - Inference Engine → Semantic Inference: dispatches generation
fish_speech/inference_engine/__init__.py - Semantic Inference → Drama Generation: uses
fish_speech/models/text2semantic/inference.py - Semantic Inference → Content Sequences: builds sequences
fish_speech/models/text2semantic/inference.py - Content Sequences → Fish Tokenizer: encodes text
fish_speech/content_sequence.py - Drama Generation → Audio Decoder Tower: decodes frames
fish_speech/models/text2semantic/fish_omni/generate.py - Reference Loader → Audio Files: reads references
fish_speech/inference_engine/reference_loader.py - Reference Loader → DAC Codec: encodes audio
fish_speech/inference_engine/reference_loader.py - VQ Manager → DAC Codec: encodes and decodes
fish_speech/inference_engine/vq_manager.py