How ahmed-ezzat20/chatterbox-finetuning-multilingual works
Chatterbox is a neural speech-generation package for English and 23-language multilingual TTS, with zero-shot voice prompting, emotion control, watermarking, and voice conversion. The main user flow is text and optional reference audio into a TTS API, then generated speech; the APIs prepare speaker and speech-prompt conditioning and use learned speech-generation components. The README also documents LoRA fine-tuning. Some model internals and API generation code were not excerpted, so uncertain runtime links are omitted or marked dashed.
Main entry points: mtl_tts.py Multilingual TTS · tts.py English TTS · t3.py T3 Generation · s3gen.py Speech Generator
Components 20
Speech APIs
- English TTS
src/chatterbox/tts.py - Multilingual TTS
src/chatterbox/mtl_tts.py - Voice Conversion
src/chatterbox/vc.py
Text And Voice Prep
- Text Tokenizers
src/chatterbox/models/tokenizers/tokenizer.py - Speech Tokenizer
src/chatterbox/models/s3tokenizer/s3tokenizer.py - Voice Encoder
src/chatterbox/models/voice_encoder/voice_encoder.py - Mel Frontend
src/chatterbox/models/voice_encoder/melspec.py
Speech Models
- T3 Generation
src/chatterbox/models/t3/t3.py - T3 Conditioning
src/chatterbox/models/t3/modules/cond_enc.py - Prompt Resampler
src/chatterbox/models/t3/modules/perceiver.py - T3 Config
src/chatterbox/models/t3/modules/t3_config.py - Llama Config
src/chatterbox/models/t3/llama_configs.py - Alignment Analysis
src/chatterbox/models/t3/inference/alignment_stream_analyzer.py - HF Inference Backend
src/chatterbox/models/t3/inference/t3_hf_backend.py - Speech Generator
src/chatterbox/models/s3gen/s3gen.py - Matcha Transformer
src/chatterbox/models/s3gen/matcha/transformer.py
Model Tuning
- LoRA Fine-Tuning
lora.py
Other components
- Caller
- Speech Dataset
- Generated Audio
Connections 25
- Caller → English TTS: requests speech
README.md - Caller → Multilingual TTS: requests speech
README.md - Caller → Voice Conversion: requests conversion
- English TTS → Generated Audio: returns speech
README.md - Multilingual TTS → Generated Audio: returns speech
README.md - English TTS → Voice Encoder: embeds reference
src/chatterbox/tts.py - English TTS → Speech Generator: embeds reference
src/chatterbox/tts.py - English TTS → Speech Tokenizer: tokenizes prompt
src/chatterbox/tts.py - Multilingual TTS → Voice Encoder: embeds reference
src/chatterbox/mtl_tts.py - Multilingual TTS → Speech Generator: embeds reference
src/chatterbox/mtl_tts.py - Multilingual TTS → Speech Tokenizer: tokenizes prompt
src/chatterbox/mtl_tts.py - English TTS → Text Tokenizers: encodes text
- Multilingual TTS → Text Tokenizers: encodes text
- Voice Encoder → Mel Frontend: uses features
src/chatterbox/models/voice_encoder/voice_encoder.py - T3 Generation → T3 Conditioning: encodes conditions
src/chatterbox/models/t3/t3.py - T3 Conditioning → Prompt Resampler: resamples prompt
src/chatterbox/models/t3/modules/cond_enc.py - T3 Generation → T3 Config: uses settings
src/chatterbox/models/t3/t3.py - T3 Generation → Llama Config: loads backbone settings
src/chatterbox/models/t3/t3.py - T3 Generation → Alignment Analysis: references analysis
src/chatterbox/models/t3/t3.py - T3 Generation → HF Inference Backend: references backend
src/chatterbox/models/t3/t3.py - Speech Generator → Matcha Transformer: uses transformer
- Voice Conversion → Speech Tokenizer: uses tokenizer
src/chatterbox/vc.py - Voice Conversion → Speech Generator: uses generator
src/chatterbox/vc.py - Speech Dataset → LoRA Fine-Tuning: provides training audio
README.md - LoRA Fine-Tuning → Multilingual TTS: loads model
lora.py