# ahteshamalik/chatterbox architecture

> Architecture diagram of the GitHub repository ahteshamalik/chatterbox, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahteshamalik/chatterbox
- Repository: https://github.com/ahteshamalik/chatterbox

## Overview

##### Project Type and Purpose
- Chatterbox TTS is a Python library/package that offers state-of-the-art text-to-speech (TTS) and voice-conversion (VC) capabilities.
- It provides example scripts (`example_tts.py`, `example_vc.py`, `example_for_mac.py`) and two Gradio apps (`gradio_tts_app.py`, `gradio_vc_app.py`) for interactive demos.
- Core functionality lives under `src/chatterbox`, with subpackages for models, tokenizers, and high-level APIs (`tts.py`, `vc.py`).

##### Top-Level Components
A. Client Layer (User Interfaces)
- Example scripts (CLI)
- Gradio demo apps (web UI)
B. API Layer
- `chatterbox.tts.ChatterboxTTS`
- `chatterbox.vc.ChatterboxVC`
C. Core Pipeline
1. Text/Audio Input
2. Tokenization (`src/chatterbox/models/s3tokenizer`)
3. Text Encoder / Llama-based backbone (`src/chatterbox/models/t3`)
4. Acoustic Model (`src/chatterbox/models/s3gen`)
– modules for flow, F0 predictor, HiFi-GAN vocoder
5. Voice Encoder (for VC) (`src/chatterbox/models/voice_encoder`)
6. Post-processing & Watermarking (`perth` integration)
D. External Dependencies
- PyTorch / torchaudio
- PERTH Watermarker
- Hugging Face Transformers (for Llama)
- Utility libs (NumPy, math, etc.)

##### Relationships and Data Flow
- Client → API (TTS/VC class methods) → Pipeline stages
a. For TTS: text → tokenizer → encoder → acoustic model → vocoder → watermark → WAV file
b. For VC: reference audio → voice encoder → style embedding → acoustic model → vocoder → WAV file
- All model components communicate via well-defined Python interfaces and configuration objects (`configs.py`, `const.py`).

##### Architectural Patterns & Design Principles
- Layered/pipeline architecture: clear separation of concerns (tokenizer vs encoder vs vocoder).
- Modular plugin approach: each model (S3Gen, T3, voice_encoder) is self-contained under its own directory.
- Configuration-driven: `configs.py` centralizes hyperparameters.
- Dependency isolation: external libraries are only imported at the layer where they’re needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        CLI_TTS["example_tts.py"]:::client
        CLI_VC["example_vc.py"]:::client
        CLI_MAC["example_for_mac.py"]:::client
        GUI_TTS["gradio_tts_app.py"]:::client
        GUI_VC["gradio_vc_app.py"]:::client
    end

    %% API Layer
    API_TTS["ChatterboxTTS"]:::core
    API_VC["ChatterboxVC"]:::core

    %% Core Library
    subgraph "Core Library"
        direction TB
        subgraph "Tokenizer"
            Token_Generic["tokenizers/tokenizer.py"]:::core
            Token_S3["s3tokenizer/s3tokenizer.py"]:::core
        end
        TextEnc["T3 Encoder"]:::core
        subgraph "Acoustic Model"
            AM_Core["s3gen.py"]:::core
            AM_Decoder["decoder.py"]:::core
            AM_F0["f0_predictor.py"]:::core
            AM_Flow["flow.py"]:::core
            AM_FlowM["flow_matching.py"]:::core
        end
        Vocoder["HiFi-GAN Vocoder"]:::core
        subgraph "Voice Encoder"
            VE_Core["voice_encoder.py"]:::core
            VE_Config["config.py"]:::core
            VE_Mel["melspec.py"]:::core
        end
        Watermark["PERTH Watermarker"]:::core
    end

    %% Data Artifacts
    TOK_Cyl["Token Stream"]:::data
    MEL_Cyl["Mel-Spectrogram"]:::data
    WAV_Cyl["WAV File"]:::data

    %% External Dependencies
    subgraph "External Dependencies"
        PyTorch["PyTorch"]:::external
        Torchaudio["torchaudio"]:::external
        HFTransformers["HuggingFace Transformers"]:::external
        Numpy["NumPy"]:::external
    end

    %% Connections Client -> API
    CLI_TTS -->|uses| API_TTS
    GUI_TTS -->|uses| API_TTS
    CLI_VC -->|uses| API_VC
    GUI_VC -->|uses| API_VC

    %% TTS Pipeline
    API_TTS -->|text input| Token_S3
    Token_S3 -->|tokens| TOK_Cyl
    TOK_Cyl -->|tokens| TextEnc
    TextEnc -->|encoded features| AM_Core
    AM_Core -->|acoustic features| MEL_Cyl
    MEL_Cyl -->|mel→waveform| Vocoder
    Vocoder -->|waveform| Watermark
    Watermark -->|watermarked audio| WAV_Cyl

    %% VC Pipeline
    API_VC -->|audio input| VE_Core
    VE_Core -->|style embedding| MEL_Cyl

    %% External dependencies usage
    API_TTS --> PyTorch
    API_VC --> PyTorch
    VO_Coder_TMP[ ] -.-> Torchaudio
    TextEnc --> HFTransformers
    AM_Core --> Numpy

    %% Click Events
    click CLI_TTS "https://github.com/ahteshamalik/chatterbox/blob/master/example_tts.py"
    click CLI_VC "https://github.com/ahteshamalik/chatterbox/blob/master/example_vc.py"
    click CLI_MAC "https://github.com/ahteshamalik/chatterbox/blob/master/example_for_mac.py"
    click GUI_TTS "https://github.com/ahteshamalik/chatterbox/blob/master/gradio_tts_app.py"
    click GUI_VC "https://github.com/ahteshamalik/chatterbox/blob/master/gradio_vc_app.py"
    click API_TTS "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/tts.py"
    click API_VC "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/vc.py"
    click Token_S3 "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3tokenizer/s3tokenizer.py"
    click Token_Generic "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/tokenizers/tokenizer.py"
    click TextEnc "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/t3/t3.py"
    click AM_Core "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3gen/s3gen.py"
    click AM_Decoder "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3gen/decoder.py"
    click AM_F0 "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3gen/f0_predictor.py"
    click AM_Flow "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3gen/flow.py"
    click AM_FlowM "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3gen/flow_matching.py"
    click Vocoder "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/s3gen/hifigan.py"
    click VE_Core "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/voice_encoder/voice_encoder.py"
    click VE_Config "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/voice_encoder/config.py"
    click VE_Mel "https://github.com/ahteshamalik/chatterbox/blob/master/src/chatterbox/models/voice_encoder/melspec.py"

    %% Styles
    classDef client fill:#D6EAF8,stroke:#1B4F72,color:#1B4F72
    classDef core fill:#D5F5E3,stroke:#196F3D,color:#196F3D
    classDef external fill:#FAD7A0,stroke:#B9770E,color:#B9770E,shape:hexagon
    classDef data fill:#F9E79F,stroke:#B7950B,color:#B7950B,shape:cylinder
```
