How resemble-ai/chatterbox works
Chatterbox is an archived family of neural speech-generation models: English and multilingual text-to-speech, Turbo/Nano low-latency TTS, and voice conversion. The principal workflow accepts text and optionally a reference voice, prepares speaker and speech-prompt conditionals, generates speech tokens, and decodes them to audio. The graph separates public model workflows from text/speaker conditioning and speech-token/audio generation. Some implementation files are indexed but unsampled, so their internal call links are omitted where the excerpts do not establish them.
Main entry points: s3gen.py Speech decoder · tts.py English TTS · mtl_tts.py Multilingual TTS · tts_turbo.py Turbo and Nano
Components 20
Speech workflows
- English TTS
src/chatterbox/tts.py - Multilingual TTS
src/chatterbox/mtl_tts.py - Turbo and Nano
src/chatterbox/tts_turbo.py - Voice conversion
src/chatterbox/vc.py
Text and conditioning
- Text tokenizers
src/chatterbox/models/tokenizers/tokenizer.py - Speaker embeddings
src/chatterbox/models/voice_encoder/voice_encoder.py - T3 conditioning
src/chatterbox/models/t3/modules/cond_enc.py - T3 configuration
src/chatterbox/models/t3/modules/t3_config.py - Speaker features
src/chatterbox/models/s3gen/xvector.py
Speech generation
- Text-to-speech model
src/chatterbox/models/t3/t3.py - T3 inference backend
src/chatterbox/models/t3/inference/t3_hf_backend.py - T3 model configs
src/chatterbox/models/t3/llama_configs.py - Speech tokenizer
src/chatterbox/models/s3tokenizer/s3tokenizer.py - Speech decoder
src/chatterbox/models/s3gen/s3gen.py - Audio parameters
src/chatterbox/models/s3gen/const.py - Matcha decoder
src/chatterbox/models/s3gen/matcha/decoder.py - Matcha transformer
src/chatterbox/models/s3gen/matcha/transformer.py - Mel text encoder
src/chatterbox/models/s3gen/matcha/text_encoder.py - HiFi-GAN audio
src/chatterbox/models/s3gen/hifigan.py
Other components
- Application user
Connections 32
- Application user → English TTS: requests speech
README.md - Application user → Multilingual TTS: requests speech
README.md - Application user → Turbo and Nano: requests speech
README.md - Application user → Voice conversion: requests conversion
README.md - English TTS → Text tokenizers: tokenizes text
src/chatterbox/tts.py - English TTS → Speaker embeddings: embeds reference
src/chatterbox/tts.py - English TTS → Speech tokenizer: encodes prompt
src/chatterbox/tts.py - English TTS → Text-to-speech model: generates tokens
src/chatterbox/tts.py - English TTS → Speech decoder: decodes tokens
src/chatterbox/tts.py - Multilingual TTS → Text tokenizers: tokenizes text
src/chatterbox/mtl_tts.py - Multilingual TTS → Speaker embeddings: embeds reference
src/chatterbox/mtl_tts.py - Multilingual TTS → Speech tokenizer: encodes prompt
src/chatterbox/mtl_tts.py - Multilingual TTS → Text-to-speech model: generates tokens
src/chatterbox/mtl_tts.py - Multilingual TTS → Speech decoder: decodes tokens
src/chatterbox/mtl_tts.py - Turbo and Nano → Text tokenizers: tokenizes text
src/chatterbox/tts_turbo.py - Turbo and Nano → Speaker embeddings: embeds reference
src/chatterbox/tts_turbo.py - Turbo and Nano → Speech tokenizer: encodes prompt
src/chatterbox/tts_turbo.py - Turbo and Nano → Text-to-speech model: generates tokens
src/chatterbox/tts_turbo.py - Turbo and Nano → Speech decoder: decodes tokens
src/chatterbox/tts_turbo.py - Voice conversion → Speech tokenizer: uses speech tokens
src/chatterbox/vc.py - Voice conversion → Speech decoder: uses audio decoder
src/chatterbox/vc.py - Text-to-speech model → T3 conditioning: encodes conditions
src/chatterbox/models/t3/t3.py - Text-to-speech model → T3 configuration: uses configuration
src/chatterbox/models/t3/t3.py - Text-to-speech model → T3 model configs: selects backbone
src/chatterbox/models/t3/t3.py - T3 conditioning → T3 configuration: reads settings
src/chatterbox/models/t3/modules/cond_enc.py - T3 conditioning → Speaker embeddings: projects speaker vector
- Speech decoder → Matcha decoder: uses decoder
- Matcha decoder → Matcha transformer: uses transformer
src/chatterbox/models/s3gen/matcha/decoder.py - Speech decoder → HiFi-GAN audio: synthesizes waveform
- Speech decoder → Mel text encoder: uses text encoder
- Speech decoder → Speaker features: uses speaker features
- Speech decoder → Audio parameters: uses audio parameters