How fishaudio/bert-vits2 works
And Label Main Components
• Input/Pre-processing Layer:
• Modules include text cleaning and tokenization and are located in the “text” directories (e.g., text/cleaner.py, text/chinese.py, text/english.py, and similar variants in oldVersion folders and onnx_modules for inference).
• Preprocess scripts like “preprocess_text.py”, “webui_preprocess.py”, and various tool scripts (e.g., tools/translate.py) support language-specific processing.
• BERT Integration and Feature Extraction:
• The “bert” directory comprises different pretrained BERT models (Japanese, Chinese, DeBERTa variants, etc.), each with its own configuration and tokenizer files.
• This component is responsible for embedding text input for downstream TTS synthesis.
• VITS2 Backbone (Acoustic Model):
• Files such as “models.py”, “losses.py”, “mel_processing.py”, “attentions.py”, and related modules indicate the core synthesis pipeline where the VITS2 model generates mel spectrograms from the processed text and BERT embeddings.
• The “config.py”, “default_config.yml”, and files in “configs” indicate model and training configurations.
• Additional Emotional/Style and Variance Modeling:
• The “emotional” directory holds models like “clap-htsat-fused” and “wav2vec2-large-robust-12-ft-emotion-msp-dim” which allow for emotional speech synthesis and additional style control.
• ONNX Inference Modules:
• The “onnx_modules” directory (with various version subdirectories such as V200, V210, V220, V230, V240, etc.) highlights the system’s support for ONNX-exported models for efficient, potentially platform-agnostic inference.
• These modules include custom implementations for attentions, model conversion, and textual components for inference.
• Deployment and UI:
• Files like “infer.py”, “onnx_infer.py”, “for_deploy/”, “webui.py”, “webui_preprocess.py”, and “hiyoriUI.py” are used to run inference and provide a user interface for model testing.
• Look at “run_MnodesAndMgpus.sh” as part of the training or deployment orchestration.
Relationships and Data Flow
• The processed text is then passed to the BERT integration layer for generating multilingual contextual embeddings.
• These embeddings, along with additional linguistic features, feed into the VITS2 backbone. Here, the acoustic model computes representations, resulting in a mel spectrogram.
• The spectrogram is then transformed into waveform audio by subsequent vocoder-like components (which might be integrated in “models.py” or further outside processing).
• For deployment, there’s an alternative path using the ONNX modules where the core synthesis pipeline is optimized for inference.
• A web UI or command-line interface (using “webui.py” and associated scripts) provides a front-end for users to interact with the TTS system.
Architectural Patterns and Design Principles
• Modularity and Layering: The project is split into clear layers (input processing, feature extraction, acoustic modeling, and output synthesis) which helps isolate concerns.
• Deployment Abstraction: Separate deployment code (in for_deploy and ONNX modules) shows a clear separation between training code and inference optimization.
Technologies, Frameworks, and Critical Files
• The BERT models and tokenizers (under “bert/”) are key to the system’s multilingual capabilities.
• The VITS2 backbone and TTS-related processing (files like “mel_processing.py”, “losses.py”, and “models.py”) are essential to generating speech.
• ONNX is critical for efficient deployment; note the various versioned folders in “onnx_modules”.
• Configuration management is supported by numerous config files (config.py, default_config.yml, and files in “configs/”).