# resemble-ai/chatterbox architecture

> Architecture diagram of the GitHub repository resemble-ai/chatterbox, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/resemble-ai/chatterbox
- Repository: https://github.com/resemble-ai/chatterbox
- Video tour (about a minute): https://gitdiagram.com/resemble-ai/chatterbox/video

## Overview

Chatterbox is an archived family of neural speech-generation models: English and multilingual text-to-speech, Turbo/Nano low-latency TTS, and voice conversion. The principal workflow accepts text and optionally a reference voice, prepares speaker and speech-prompt conditionals, generates speech tokens, and decodes them to audio. The graph separates public model workflows from text/speaker conditioning and speech-token/audio generation. Some implementation files are indexed but unsampled, so their internal call links are omitted where the excerpts do not establish them.

Main entry points: [`src/chatterbox/models/s3gen/s3gen.py`](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/s3gen.py) (Speech decoder), [`src/chatterbox/tts.py`](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/tts.py) (English TTS), [`src/chatterbox/mtl_tts.py`](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/mtl_tts.py) (Multilingual TTS), [`src/chatterbox/tts_turbo.py`](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/tts_turbo.py) (Turbo and Nano)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Speech workflows"]
  node_english["English TTS<br/>[tts.py]"]
  node_multilingual["Multilingual TTS<br/>[mtl_tts.py]"]
  node_turbo_nano["Turbo and Nano<br/>[tts_turbo.py]"]
  node_voice_conversion["Voice conversion<br/>[vc.py]"]
end

subgraph group_conditioning["Text and conditioning"]
  node_text_tokenizer["Text tokenizers<br/>[tokenizer.py]"]
  node_speaker_encoder["Speaker embeddings<br/>[voice_encoder.py]"]
  node_conditioning_encoder["T3 conditioning<br/>[cond_enc.py]"]
  node_t3_config["T3 configuration<br/>[t3_config.py]"]
  node_xvector["Speaker features<br/>[xvector.py]"]
end

subgraph group_generation["Speech generation"]
  node_t3_model["Text-to-speech model<br/>[t3.py]"]
  node_t3_backend["T3 inference backend<br/>[t3_hf_backend.py]"]
  node_t3_architecture["T3 model configs<br/>[llama_configs.py]"]
  node_speech_tokenizer["Speech tokenizer<br/>[s3tokenizer.py]"]
  node_s3gen["Speech decoder<br/>[s3gen.py]"]
  node_s3gen_constants["Audio parameters<br/>[const.py]"]
  node_matcha_decoder["Matcha decoder<br/>[decoder.py]"]
  node_matcha_transformer["Matcha transformer<br/>[transformer.py]"]
  node_mel_encoder["Mel text encoder<br/>[text_encoder.py]"]
  node_hifigan["HiFi-GAN audio<br/>[hifigan.py]"]
end

node_user(("Application user"))

node_user -->|"requests speech"| node_english
node_user -->|"requests speech"| node_multilingual
node_user -->|"requests speech"| node_turbo_nano
node_user -->|"requests conversion"| node_voice_conversion
node_english -->|"tokenizes text"| node_text_tokenizer
node_english -->|"embeds reference"| node_speaker_encoder
node_english -->|"encodes prompt"| node_speech_tokenizer
node_english -->|"generates tokens"| node_t3_model
node_english -->|"decodes tokens"| node_s3gen
node_multilingual -->|"tokenizes text"| node_text_tokenizer
node_multilingual -->|"embeds reference"| node_speaker_encoder
node_multilingual -->|"encodes prompt"| node_speech_tokenizer
node_multilingual -->|"generates tokens"| node_t3_model
node_multilingual -->|"decodes tokens"| node_s3gen
node_turbo_nano -->|"tokenizes text"| node_text_tokenizer
node_turbo_nano -->|"embeds reference"| node_speaker_encoder
node_turbo_nano -->|"encodes prompt"| node_speech_tokenizer
node_turbo_nano -->|"generates tokens"| node_t3_model
node_turbo_nano -->|"decodes tokens"| node_s3gen
node_voice_conversion -->|"uses speech tokens"| node_speech_tokenizer
node_voice_conversion -->|"uses audio decoder"| node_s3gen
node_t3_model -->|"encodes conditions"| node_conditioning_encoder
node_t3_model -->|"uses configuration"| node_t3_config
node_t3_model -->|"selects backbone"| node_t3_architecture
node_conditioning_encoder -->|"reads settings"| node_t3_config
node_conditioning_encoder -.->|"projects speaker vector"| node_speaker_encoder
node_s3gen -.->|"uses decoder"| node_matcha_decoder
node_matcha_decoder -->|"uses transformer"| node_matcha_transformer
node_s3gen -.->|"synthesizes waveform"| node_hifigan
node_s3gen -.->|"uses text encoder"| node_mel_encoder
node_s3gen -.->|"uses speaker features"| node_xvector
node_s3gen -.->|"uses audio parameters"| node_s3gen_constants

click node_english "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/tts.py"
click node_multilingual "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/mtl_tts.py"
click node_turbo_nano "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/tts_turbo.py"
click node_voice_conversion "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/vc.py"
click node_text_tokenizer "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/tokenizers/tokenizer.py"
click node_speaker_encoder "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/voice_encoder/voice_encoder.py"
click node_conditioning_encoder "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/modules/cond_enc.py"
click node_t3_config "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/modules/t3_config.py"
click node_t3_model "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/t3.py"
click node_t3_backend "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/inference/t3_hf_backend.py"
click node_t3_architecture "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/llama_configs.py"
click node_speech_tokenizer "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3tokenizer/s3tokenizer.py"
click node_s3gen "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/s3gen.py"
click node_s3gen_constants "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/const.py"
click node_matcha_decoder "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/matcha/decoder.py"
click node_matcha_transformer "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/matcha/transformer.py"
click node_mel_encoder "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/matcha/text_encoder.py"
click node_hifigan "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/hifigan.py"
click node_xvector "https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/xvector.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_english,node_multilingual,node_turbo_nano,node_voice_conversion,node_user toneBlue
class node_text_tokenizer,node_speaker_encoder,node_conditioning_encoder,node_t3_config,node_xvector toneAmber
class node_t3_model,node_t3_backend,node_t3_architecture,node_speech_tokenizer,node_s3gen,node_s3gen_constants,node_matcha_decoder,node_matcha_transformer,node_mel_encoder,node_hifigan toneMint
```

## Components

### Speech workflows

- [English TTS](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/tts.py): `src/chatterbox/tts.py`
- [Multilingual TTS](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/mtl_tts.py): `src/chatterbox/mtl_tts.py`
- [Turbo and Nano](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/tts_turbo.py): `src/chatterbox/tts_turbo.py`
- [Voice conversion](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/vc.py): `src/chatterbox/vc.py`

### Text and conditioning

- [Text tokenizers](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/tokenizers/tokenizer.py): `src/chatterbox/models/tokenizers/tokenizer.py`
- [Speaker embeddings](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/voice_encoder/voice_encoder.py): `src/chatterbox/models/voice_encoder/voice_encoder.py`
- [T3 conditioning](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/modules/cond_enc.py): `src/chatterbox/models/t3/modules/cond_enc.py`
- [T3 configuration](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/modules/t3_config.py): `src/chatterbox/models/t3/modules/t3_config.py`
- [Speaker features](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/xvector.py): `src/chatterbox/models/s3gen/xvector.py`

### Speech generation

- [Text-to-speech model](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/t3.py): `src/chatterbox/models/t3/t3.py`
- [T3 inference backend](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/inference/t3_hf_backend.py): `src/chatterbox/models/t3/inference/t3_hf_backend.py`
- [T3 model configs](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/t3/llama_configs.py): `src/chatterbox/models/t3/llama_configs.py`
- [Speech tokenizer](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3tokenizer/s3tokenizer.py): `src/chatterbox/models/s3tokenizer/s3tokenizer.py`
- [Speech decoder](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/s3gen.py): `src/chatterbox/models/s3gen/s3gen.py`
- [Audio parameters](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/const.py): `src/chatterbox/models/s3gen/const.py`
- [Matcha decoder](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/matcha/decoder.py): `src/chatterbox/models/s3gen/matcha/decoder.py`
- [Matcha transformer](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/matcha/transformer.py): `src/chatterbox/models/s3gen/matcha/transformer.py`
- [Mel text encoder](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/matcha/text_encoder.py): `src/chatterbox/models/s3gen/matcha/text_encoder.py`
- [HiFi-GAN audio](https://github.com/resemble-ai/chatterbox/blob/master/src/chatterbox/models/s3gen/hifigan.py): `src/chatterbox/models/s3gen/hifigan.py`

### Other components

- **Application user**

## Connections

- Application user → English TTS: requests speech (evidence: `README.md`)
- Application user → Multilingual TTS: requests speech (evidence: `README.md`)
- Application user → Turbo and Nano: requests speech (evidence: `README.md`)
- Application user → Voice conversion: requests conversion (evidence: `README.md`)
- English TTS → Text tokenizers: tokenizes text (evidence: `src/chatterbox/tts.py`)
- English TTS → Speaker embeddings: embeds reference (evidence: `src/chatterbox/tts.py`)
- English TTS → Speech tokenizer: encodes prompt (evidence: `src/chatterbox/tts.py`)
- English TTS → Text-to-speech model: generates tokens (evidence: `src/chatterbox/tts.py`)
- English TTS → Speech decoder: decodes tokens (evidence: `src/chatterbox/tts.py`)
- Multilingual TTS → Text tokenizers: tokenizes text (evidence: `src/chatterbox/mtl_tts.py`)
- Multilingual TTS → Speaker embeddings: embeds reference (evidence: `src/chatterbox/mtl_tts.py`)
- Multilingual TTS → Speech tokenizer: encodes prompt (evidence: `src/chatterbox/mtl_tts.py`)
- Multilingual TTS → Text-to-speech model: generates tokens (evidence: `src/chatterbox/mtl_tts.py`)
- Multilingual TTS → Speech decoder: decodes tokens (evidence: `src/chatterbox/mtl_tts.py`)
- Turbo and Nano → Text tokenizers: tokenizes text (evidence: `src/chatterbox/tts_turbo.py`)
- Turbo and Nano → Speaker embeddings: embeds reference (evidence: `src/chatterbox/tts_turbo.py`)
- Turbo and Nano → Speech tokenizer: encodes prompt (evidence: `src/chatterbox/tts_turbo.py`)
- Turbo and Nano → Text-to-speech model: generates tokens (evidence: `src/chatterbox/tts_turbo.py`)
- Turbo and Nano → Speech decoder: decodes tokens (evidence: `src/chatterbox/tts_turbo.py`)
- Voice conversion → Speech tokenizer: uses speech tokens (evidence: `src/chatterbox/vc.py`)
- Voice conversion → Speech decoder: uses audio decoder (evidence: `src/chatterbox/vc.py`)
- Text-to-speech model → T3 conditioning: encodes conditions (evidence: `src/chatterbox/models/t3/t3.py`)
- Text-to-speech model → T3 configuration: uses configuration (evidence: `src/chatterbox/models/t3/t3.py`)
- Text-to-speech model → T3 model configs: selects backbone (evidence: `src/chatterbox/models/t3/t3.py`)
- T3 conditioning → T3 configuration: reads settings (evidence: `src/chatterbox/models/t3/modules/cond_enc.py`)
- T3 conditioning → Speaker embeddings: projects speaker vector
- Speech decoder → Matcha decoder: uses decoder
- Matcha decoder → Matcha transformer: uses transformer (evidence: `src/chatterbox/models/s3gen/matcha/decoder.py`)
- Speech decoder → HiFi-GAN audio: synthesizes waveform
- Speech decoder → Mel text encoder: uses text encoder
- Speech decoder → Speaker features: uses speaker features
- Speech decoder → Audio parameters: uses audio parameters
