# fishaudio/bert-vits2 architecture

> Architecture diagram of the GitHub repository fishaudio/bert-vits2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fishaudio/bert-vits2
- Repository: https://github.com/fishaudio/bert-vits2

## Overview

##### And Label Main Components
- Input/Pre-processing Layer:
- Modules include text cleaning and tokenization and are located in the “text” directories (e.g., text/cleaner.py, text/chinese.py, text/english.py, and similar variants in oldVersion folders and onnx_modules for inference).
- Preprocess scripts like “preprocess_text.py”, “webui_preprocess.py”, and various tool scripts (e.g., tools/translate.py) support language-specific processing.

- BERT Integration and Feature Extraction:
- The “bert” directory comprises different pretrained BERT models (Japanese, Chinese, DeBERTa variants, etc.), each with its own configuration and tokenizer files.
- This component is responsible for embedding text input for downstream TTS synthesis.

- VITS2 Backbone (Acoustic Model):
- Files such as “models.py”, “losses.py”, “mel_processing.py”, “attentions.py”, and related modules indicate the core synthesis pipeline where the VITS2 model generates mel spectrograms from the processed text and BERT embeddings.
- The “config.py”, “default_config.yml”, and files in “configs” indicate model and training configurations.

- Additional Emotional/Style and Variance Modeling:
- The “emotional” directory holds models like “clap-htsat-fused” and “wav2vec2-large-robust-12-ft-emotion-msp-dim” which allow for emotional speech synthesis and additional style control.

- ONNX Inference Modules:
- The “onnx_modules” directory (with various version subdirectories such as V200, V210, V220, V230, V240, etc.) highlights the system’s support for ONNX-exported models for efficient, potentially platform-agnostic inference.
- These modules include custom implementations for attentions, model conversion, and textual components for inference.

- Deployment and UI:
- Files like “infer.py”, “onnx_infer.py”, “for_deploy/”, “webui.py”, “webui_preprocess.py”, and “hiyoriUI.py” are used to run inference and provide a user interface for model testing.
- Look at “run_MnodesAndMgpus.sh” as part of the training or deployment orchestration.

##### Relationships and Data Flow
- The processed text is then passed to the BERT integration layer for generating multilingual contextual embeddings.
- These embeddings, along with additional linguistic features, feed into the VITS2 backbone. Here, the acoustic model computes representations, resulting in a mel spectrogram.
- The spectrogram is then transformed into waveform audio by subsequent vocoder-like components (which might be integrated in “models.py” or further outside processing).
- For deployment, there’s an alternative path using the ONNX modules where the core synthesis pipeline is optimized for inference.
- A web UI or command-line interface (using “webui.py” and associated scripts) provides a front-end for users to interact with the TTS system.

##### Architectural Patterns and Design Principles
- Modularity and Layering: The project is split into clear layers (input processing, feature extraction, acoustic modeling, and output synthesis) which helps isolate concerns.
- Deployment Abstraction: Separate deployment code (in for_deploy and ONNX modules) shows a clear separation between training code and inference optimization.

##### Technologies, Frameworks, and Critical Files
- The BERT models and tokenizers (under “bert/”) are key to the system’s multilingual capabilities.
- The VITS2 backbone and TTS-related processing (files like “mel_processing.py”, “losses.py”, and “models.py”) are essential to generating speech.
- ONNX is critical for efficient deployment; note the various versioned folders in “onnx_modules”.
- Configuration management is supported by numerous config files (config.py, default_config.yml, and files in “configs/”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    A("Raw Text Input"):::input

    subgraph "Input/Pre-processing Layer"
        T1("Cleaner (cleaner.py)"):::preproc
        T2("Chinese Tokenizer (chinese.py)"):::preproc
        T3("English Tokenizer (english.py)"):::preproc
        T4("Japanese Tokenizer (japanese.py)"):::preproc
        T5("BERT Utils (bert_utils.py)"):::preproc
        T6("Preprocess Script (preprocess_text.py)"):::preproc
        T7("WebUI Preprocess (webui_preprocess.py)"):::preproc
        T8("Translate Tool (tools/translate.py)"):::preproc
        T9("Legacy Pre-Processing (oldVersion/ ...)"):::preproc
    end

    subgraph "BERT Integration & Feature Extraction"
        BERT1("BERT Models (bert/ directory)"):::bert
        BERT2("BERT Generator (bert_gen.py)"):::bert
    end

    subgraph "VITS2 Backbone (Acoustic Model)"
        V1("Core Model Files (models.py, losses.py, mel_processing.py, attentions.py, modules.py)"):::acoustic
        V2("Configuration Files (config.py, default_config.yml)"):::acoustic
        V3("Configs Directory (configs/)"):::acoustic
        E("Emotional/Style Module (emotional/ directory)"):::emotive
    end

    subgraph "ONNX Inference Modules"
        O1("ONNX Models (onnx_modules/ directory)"):::onnx
        O2("ONNX Inference Script (onnx_infer.py)"):::onnx
        O3("ONNX Initialization (__init__.py)"):::onnx
    end

    subgraph "Deployment & UI"
        D1("Deployment Scripts (for_deploy/infer.py, infer_utils.py, for_deploy/webui.py)"):::deploy
        D2("User Interface (webui.py, hiyoriUI.py)"):::deploy
        D3("Orchestration (run_MnodesAndMgpus.sh)"):::deploy
    end

    F("Synthesized Mel Spectrogram"):::output
    G("Waveform Audio Output"):::output

    %% Data Flow Connections
    A --> T1
    T9 --> BERT1
    BERT1 --> BERT2
    BERT2 --> V1
    E --> V1
    V1 --> V2
    V1 --> V3
    V1 --> F
    F --> G
    %% ONNX branch from the acoustic core
    V1 ---> O1
    O1 --> O2
    O2 --> O3
    %% Merge inference outputs into deployment
    G --> D1
    O3 --> D1
    D1 --> D2
    D1 --> D3

    %% Click Events
    click T1 "https://github.com/fishaudio/bert-vits2/blob/master/text/cleaner.py"
    click T2 "https://github.com/fishaudio/bert-vits2/blob/master/text/chinese.py"
    click T3 "https://github.com/fishaudio/bert-vits2/blob/master/text/english.py"
    click T4 "https://github.com/fishaudio/bert-vits2/blob/master/text/japanese.py"
    click T5 "https://github.com/fishaudio/bert-vits2/blob/master/text/bert_utils.py"
    click T6 "https://github.com/fishaudio/bert-vits2/blob/master/preprocess_text.py"
    click T7 "https://github.com/fishaudio/bert-vits2/blob/master/webui_preprocess.py"
    click T8 "https://github.com/fishaudio/bert-vits2/blob/master/tools/translate.py"
    click T9 "https://github.com/fishaudio/bert-vits2/tree/master/oldVersion/V101/text/"
    click BERT1 "https://github.com/fishaudio/bert-vits2/tree/master/bert/"
    click BERT2 "https://github.com/fishaudio/bert-vits2/blob/master/bert_gen.py"
    click V1 "https://github.com/fishaudio/bert-vits2/blob/master/models.py"
    click V2 "https://github.com/fishaudio/bert-vits2/blob/master/config.py"
    click V3 "https://github.com/fishaudio/bert-vits2/tree/master/configs/"
    click E "https://github.com/fishaudio/bert-vits2/tree/master/emotional/"
    click O1 "https://github.com/fishaudio/bert-vits2/tree/master/onnx_modules/"
    click O2 "https://github.com/fishaudio/bert-vits2/blob/master/onnx_infer.py"
    click O3 "https://github.com/fishaudio/bert-vits2/blob/master/onnx_modules/__init__.py"
    click D1 "https://github.com/fishaudio/bert-vits2/blob/master/for_deploy/infer.py"
    click D2 "https://github.com/fishaudio/bert-vits2/blob/master/webui.py"
    click D3 "https://github.com/fishaudio/bert-vits2/blob/master/run_MnodesAndMgpus.sh"

    %% Styles
    classDef input fill:#f9f,stroke:#333,stroke-width:2px;
    classDef preproc fill:#ccf,stroke:#333,stroke-width:2px;
    classDef bert fill:#cfc,stroke:#333,stroke-width:2px;
    classDef acoustic fill:#fc9,stroke:#333,stroke-width:2px;
    classDef emotive fill:#ff9,stroke:#333,stroke-width:2px;
    classDef onnx fill:#9fc,stroke:#333,stroke-width:2px;
    classDef deploy fill:#f99,stroke:#333,stroke-width:2px;
    classDef output fill:#9cf,stroke:#333,stroke-width:2px;

    class A input;
    class T1,T2,T3,T4,T5,T6,T7,T8,T9 preproc;
    class BERT1,BERT2 bert;
    class V1,V2,V3 acoustic;
    class E emotive;
    class O1,O2,O3 onnx;
    class F,G output;
    class D1,D2,D3 deploy;
```
