# yomio-tech-inc/gpt-sovits2 architecture

> Architecture diagram of the GitHub repository yomio-tech-inc/gpt-sovits2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yomio-tech-inc/gpt-sovits2
- Repository: https://github.com/yomio-tech-inc/gpt-sovits2

## Overview

Based on our analysis of both the provided file tree and README, the project is an advanced text-to-speech (TTS) system—GPT-SoVITS2—that extends the original GPT-SoVITS framework. It integrates state‐of‐the‐art pre-trained language and speech models (such as BGE-M3, mHubert, and Qwen2-0.3b), and it notably supports multi-language processing, zero-shot voice adaptation, and even voice style fusion. The project has components for data processing, feature extraction, model training (both autoregressive and VITS-related parts), and various utilities and tools (including Docker and deployment/webapi scripts) to facilitate development and inference.

##### Main Components
- Data Processing:
– Directories like “GPT_SoVITS/AR/data” and “GPT_SoVITS/prepare_datasets” suggest modules for dataset preparation, sampling, preprocessing, and batching.
- Text Processing & Tokenization:
– Modules under “GPT_SoVITS/AR/text_processing” and “GPT_SoVITS/text” handle transformation from raw text into a BPE-tokenized format for embedding and further processing.
- Feature Extraction:
– The “GPT_SoVITS/feature_extractor” directory contains scripts (e.g., “hubert.py”, “whisper_enc.py”) used to extract embeddings from raw audio.
- Autoregressive (AR) Model / GPT Core:
– Found under “GPT_SoVITS/AR/models” and “GPT_SoVITS/AR/modules”—this is the core autoregressive transformer block (with modifications such as the Qwen2-0.3b upgrade) integrating the text and speech embeddings.
- Speech Decoder:
– The speech synthesis component based on VITS/VITS2 is integrated within “GPT_SoVITS/module” and related scripts (like “s1_train.py”, “s2_train.py”) and supports transformer blocks and additional quantization.
- Inference Interfaces:
– Scripts like “inference_gui.py”, “inference_webui.py”, and notebooks in “playground” indicate multiple entry points for testing and demonstration (including API endpoints, web UI, and interactive notebooks).
- Docker & Deployment:
– The presence of Dockerfiles, docker-compose.yaml, and related deployment scripts ensure that the system is containerized for ease of deployment.

##### Relationships and Interactions
- Data flows from raw audio/text inputs (via dataset preparation modules) through text processing and feature extraction.
- Both text and audio features are embedded separately (using models like BGE-M3 for text and mHubert or $S^3$ Encoder for speech) and then fed into the AR model.
- The AR model (with Qwen2-0.3b architecture) takes these embedded tokens and produces a latent representation that bridges to the speech decoder.
- The speech decoder (VITS2) then takes the latent sequence and produces the final waveform output.
- Additionally, supplementary components (like reference audio for style fusion) allow multiple input streams, influencing the embedding and synthesis pipelines.
- Tools and utilities (in the “tools” and “tools/uvr5” directories) aid in tasks such as denoising, slicing audio, and internationalization (i18n).

##### Architectural Patterns & Design Principles
- Modularity: The code is divided into distinct modules (data, models, utils, feature_extractor) that encapsulate specific responsibilities.
- Pipeline Architecture: There is a clear sequential pipeline (data ingestion → tokenization/embedding → autoregressive transformation → decoding) that can be visually mapped.
- Extensibility: The presence of configuration files (e.g., YAML and JSON) and multiple language folders (for i18n) suggests the design intends for easy modification and extension.
- Separation of Concerns: Training scripts, inference GUIs, and deployment tools are separated into distinct directories for clarity.

##### Technologies & Frameworks to Highlight
- Deep Learning: Use of PyTorch (implied by Lightning modules and typical training/inference scripts in TTS systems).
- Docker & Containerization: The Dockerfiles, docker-compose, and development container configuration indicate an emphasis on reproducible environments.
- Notebooks & Scripting: Jupyter notebooks for experimentation (e.g., “GPT_SoVITS_Inference.ipynb”, “playground/*.ipynb”).
- Pre-trained Models: Integration of models like mHubert, Qwen2-0.3b, BGE-M3 which are central to the system’s functionality.
- Internationalization: The “i18n” directory with multiple locale JSON files.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Layer
    subgraph "1. Data & Preprocessing"
        DP1["Data (AR/data)"]
        DP2["Data (prepare_datasets)"]
        DI["Data Input"]
    end

    %% Text Processing & Tokenization Layer
    subgraph "2. Text Processing & Tokenization"
        TP1["Text (AR/text_processing)"]
        TP2["Text (text)"]
        TE["Text Embedding"]
    end

    %% Audio Feature Extraction Layer
    subgraph "3. Feature Extraction"
        FE["Audio Feature Extraction (feature_extractor)"]
        FE1["hubert.py"]
        FE2["whisper_enc.py"]
        AE["Audio Embedding"]
    end

    %% Autoregressive Core Model Layer
    subgraph "4. Autoregressive Core Model"
        AR1["AR Models"]
        AR2["AR Modules"]
    end

    %% Speech Decoder Layer
    subgraph "5. Speech Decoder"
        SD["Speech Decoder (VITS2)"]
        S1["s1_train.py"]
        S2["s2_train.py"]
    end

    %% Output Generation Layer
    subgraph "6. Output Generation"
        OUT["Waveform Synthesis"]
    end

    %% Inference Interfaces Layer
    subgraph "7. Inference Interfaces"
        IG1["inference_gui.py"]
        IW1["inference_webui.py"]
        PN["Playground Notebooks"]
    end

    %% Deployment & Supporting Tools (Side Components)
    subgraph "A. Deployment & Supporting Tools"
        subgraph "Docker & Deployment"
            DF["Dockerfile"]
            DC["docker-compose.yaml"]
            DEV[".devcontainer"]
            DOCK["Docker Directory"]
        end
        subgraph "Tools & Configurations"
            T["Tools & Utilities"]
            CFG1["GPT_SoVITS/configs"]
            CFG2["configs"]
        end
    end

    %% Data Flow Connections
    DP1 --> DI
    DP2 --> DI
    DI --> TP1
    DI --> FE

    TP1 --> TE
    TP2 --> TE

    FE --> AE
    FE1 --> FE
    FE2 --> FE

    TE --> AR1
    AE --> AR1

    AR1 --> AR2

    AR2 --> SD

    SD --> S1
    SD --> S2

    SD --> OUT

    OUT --> IG1
    OUT --> IW1
    OUT --> PN

    %% Side/Supporting Connections (dashed lines)
    CFG1 --- DI
    CFG2 --- TE
    T --- DP1
    T --- TP1
    T --- FE
    DF -.-> OUT
    DC -.-> OUT
    DEV -.-> OUT
    DOCK -.-> OUT

    %% Styling: Colors and Classes
    class DP1,DP2,DI data
    class TP1,TP2,TE text
    class FE,FE1,FE2,AE feature
    class AR1,AR2 core
    class SD,S1,S2 decoder
    class OUT output
    class IG1,IW1,PN inference
    class DF,DC,DEV,DOCK deployment
    class T,CFG1,CFG2 tools

    %% Class Definitions
    classDef data fill:#f2d7d5,stroke:#bd362f,stroke-width:2px;
    classDef text fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef feature fill:#d6eaf8,stroke:#2e86c1,stroke-width:2px;
    classDef core fill:#fcf3cf,stroke:#f0ad4e,stroke-width:2px;
    classDef decoder fill:#e8daef,stroke:#8e44ad,stroke-width:2px;
    classDef output fill:#d5dbdb,stroke:#7f8c8d,stroke-width:2px;
    classDef inference fill:#f9e79f,stroke:#f39c12,stroke-width:2px;
    classDef deployment fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef tools fill:#d6dbdf,stroke:#34495e,stroke-width:2px;

    %% Click Events
    click DP1 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/AR/data"
    click DP2 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/prepare_datasets"
    click TP1 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/AR/text_processing"
    click TP2 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/text"
    click FE "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/feature_extractor"
    click FE1 "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/GPT_SoVITS/feature_extractor/hubert.py"
    click FE2 "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/GPT_SoVITS/feature_extractor/whisper_enc.py"
    click AR1 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/AR/models"
    click AR2 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/AR/modules"
    click SD "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/module"
    click S1 "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/GPT_SoVITS/module/s1_train.py"
    click S2 "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/GPT_SoVITS/module/s2_train.py"
    click IG1 "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/GPT_SoVITS/inference_gui.py"
    click IW1 "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/GPT_SoVITS/inference_webui.py"
    click PN "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/playground"
    click DF "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/Dockerfile"
    click DC "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/docker-compose.yaml"
    click DEV "https://github.com/yomio-tech-inc/gpt-sovits2/blob/main/.devcontainer"
    click DOCK "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/Docker Directory"
    click CFG1 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/GPT_SoVITS/configs"
    click CFG2 "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/configs"
    click T "https://github.com/yomio-tech-inc/gpt-sovits2/tree/main/tools"
```
