# jaywalnut310/vits architecture

> Architecture diagram of the GitHub repository jaywalnut310/vits, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jaywalnut310/vits
- Repository: https://github.com/jaywalnut310/vits

## Overview

##### Project Type and Purpose
- VITS is an end-to-end text-to-speech (TTS) system: a single-stage, parallelizable neural TTS model combining variational inference (VAE), normalizing flows, and adversarial training.
- Its primary purpose is to convert raw text into natural-sounding speech with diverse prosody.

##### File Structure
- Top-level Python scripts:
– train.py & train_ms.py: launch the training pipelines for single-speaker (LJ) and multi-speaker (VCTK).
– inference.ipynb: demonstrates the inference pipeline.
- Core modules (modules.py, models.py, attentions.py, commons.py, losses.py, transforms.py, mel_processing.py, data_utils.py, utils.py): implement encoders, decoders, flows, discriminators, feature transforms, loss functions, data loaders.
- preprocess.py & monotonic_align/: the G2P-and-duration preprocessing package (Cython-accelerated alignment).
- configs/*.json: hyperparameter definitions for different datasets.
- filelists/: train/val/test splits for LJ and VCTK.
- text/: text cleaning & symbol definitions.
- resources/: embedded architecture figures (training vs. inference).

##### README for Insights
- The text emphasizes:
– Encoder (text → posterior encoder + normalizing flows)
– Stochastic duration predictor
– Generator (decoder)
– Discriminators (multi-scale, multi-period)
– VAE latent sampling (KL loss)
– GAN-style adversarial loss
– Separate training vs. inference data flow.

##### Main Components & Their Relationships
A. Data Ingestion & Preprocessing
- Raw text & waveform datasets (LJ Speech, VCTK)
- Text cleaning (text/cleaners.py), G2P, phoneme indexing
- Monotonic alignment search (monotonic_align): align text with mel frames
- Mel-spectrogram extraction (mel_processing.py)
- Filelists drive batching.
B. Model Architecture (Training Mode)
1. Text Encoder → produces hidden text features
2. Posterior Encoder (mel→latent) + Normalizing Flows → posterior latent z
3. Stochastic Duration Predictor → predicts durations for text frames
4. Generator (Decoder) → converts z to waveform
5. Discriminators → adversarial feedback
6. Losses: reconstruction (L1/L2), KL divergence, adversarial, duration, feature matching
C. Model Architecture (Inference Mode)
1. Text Encoder → text features
2. Stochastic Duration Predictor → durations
3. Prior sampling + Normalizing Flows inversely → latent z
4. Generator → waveform output
D. Utilities & Scripts
- train.py / train_ms.py orchestrate data preload, model instantiation, optimization loops
- Inference notebook runs the above inference pipeline
- configs/*.json supply hyperparameters to scripts.

##### Architectural Patterns & Technologies
- Pattern: VAE + Normalizing Flows + GAN (adversarial training) in a single unified network.
- Modularization: clear separation of text frontend, posterior/prior networks, duration model, generator, discriminators.
- Data pipeline: Python + PyTorch DataLoaders, Cython for alignment speedup.
- CLI/config-driven experimentation (JSON configs, argparse).
- Dependencies: Python ≥3.6, PyTorch, librosa, numpy, Cython.

##### Final Checklist
- Two panels: Training vs. Inference
- Clear labels for:
– Data sources (LJ / VCTK wav folders)
– Preprocessing modules
– Neural network components (encoder, flow, decoder, discriminator)
– Scripts/orchestration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources & Preprocessing
    RawAudio(("Raw Audio (WAVs)")):::data
    Filelists(("Filelists")):::data
    Cleaning(("Text Cleaning & Symbols")):::pre
    Alignment(("G2P & Monotonic Align")):::pre
    MelProc(("Mel-Spectrogram Extraction")):::pre
    DataLoaders(("Data Loaders")):::utils

    RawAudio --> Cleaning
    Cleaning --> Alignment
    Alignment --> MelProc
    MelProc --> DataLoaders
    Filelists --> DataLoaders

    %% Training Mode
    subgraph "Training Mode"
        TextEnc(("Text Encoder")):::nn
        Posterior(("Posterior Encoder + Normalizing Flows")):::nn
        DurationPred(("Stochastic Duration Predictor")):::nn
        Generator(("Generator (Decoder)")):::nn
        Discriminators(("Multi-Scale/Period Discriminators")):::loss
        Losses(("Losses: KL, Adversarial, L1, Duration, Feature-Matching")):::loss
        CoachTrain(("train.py / train_ms.py")):::script

        DataLoaders --> TextEnc
        DataLoaders --> Posterior
        TextEnc --> Posterior
        Posterior --> Generator
        Posterior --> DurationPred
        DurationPred --> Generator
        Generator --> Discriminators
        Generator --> Losses
        Discriminators --> Losses
        Losses --> CoachTrain
    end

    %% Inference Mode
    subgraph "Inference Mode"
        TextEnc_I(("Text Encoder")):::nn
        DurationPred_I(("Stochastic Duration Predictor")):::nn
        PriorFlow(("Prior Sampling + Inverse Flows")):::nn
        Generator_I(("Generator (Decoder)")):::nn
        InfScript(("inference.ipynb")):::script

        DataLoaders --> TextEnc_I
        TextEnc_I --> DurationPred_I
        DurationPred_I --> PriorFlow
        PriorFlow --> Generator_I
        Generator_I --> InfScript
    end

    %% Configuration & Utilities
    Configs(("Config JSONs")):::utils
    PreprocessScript(("Preprocess Script")):::utils
    Commons(("Commons & Transforms")):::utils
    Utils(("Helper Functions & Logging")):::utils
    Resources(("README, LICENSE & Figures")):::utils

    Configs -.-> CoachTrain
    Configs -.-> InfScript
    PreprocessScript -.-> Cleaning
    Commons -.-> TextEnc
    Commons -.-> Posterior
    Commons -.-> DurationPred
    Commons -.-> Generator
    Commons -.-> Discriminators
    Utils -.-> CoachTrain
    Utils -.-> InfScript

    %% Click Events
    click Cleaning "https://github.com/jaywalnut310/vits/blob/main/text/cleaners.py"
    click Cleaning "https://github.com/jaywalnut310/vits/blob/main/text/symbols.py"
    click Alignment "https://github.com/jaywalnut310/vits/tree/main/monotonic_align/"
    click MelProc "https://github.com/jaywalnut310/vits/blob/main/mel_processing.py"
    click DataLoaders "https://github.com/jaywalnut310/vits/blob/main/data_utils.py"
    click TextEnc "https://github.com/jaywalnut310/vits/blob/main/modules.py"
    click Posterior "https://github.com/jaywalnut310/vits/blob/main/modules.py"
    click DurationPred "https://github.com/jaywalnut310/vits/blob/main/modules.py"
    click Generator "https://github.com/jaywalnut310/vits/blob/main/modules.py"
    click Discriminators "https://github.com/jaywalnut310/vits/blob/main/modules.py"
    click Posterior "https://github.com/jaywalnut310/vits/blob/main/transforms.py"
    click Commons "https://github.com/jaywalnut310/vits/blob/main/commons.py"
    click PriorFlow "https://github.com/jaywalnut310/vits/blob/main/transforms.py"
    click CoachTrain "https://github.com/jaywalnut310/vits/blob/main/train.py"
    click CoachTrain "https://github.com/jaywalnut310/vits/blob/main/train_ms.py"
    click InfScript "https://github.com/jaywalnut310/vits/blob/main/inference.ipynb"
    click Configs "https://github.com/jaywalnut310/vits/blob/main/configs/ljs_base.json"
    click Configs "https://github.com/jaywalnut310/vits/blob/main/configs/ljs_nosdp.json"
    click Configs "https://github.com/jaywalnut310/vits/blob/main/configs/vctk_base.json"
    click PreprocessScript "https://github.com/jaywalnut310/vits/blob/main/preprocess.py"
    click Utils "https://github.com/jaywalnut310/vits/blob/main/utils.py"
    click Resources "https://github.com/jaywalnut310/vits/blob/main/README.md"
    click Resources "https://github.com/jaywalnut310/vits/tree/main/LICENSE"
    click Resources "https://github.com/jaywalnut310/vits/blob/main/resources/fig_1a.png"
    click Resources "https://github.com/jaywalnut310/vits/blob/main/resources/fig_1b.png"

    %% Styles
    classDef data fill:#D6EAF8,stroke:#5DADE2,color:#1B4F72
    classDef pre fill:#EBF5FB,stroke:#85C1E9,color:#1B2631
    classDef nn fill:#D5F5E3,stroke:#52BE80,color:#145A32
    classDef utils fill:#FDEBD0,stroke:#F5CBA7,color:#87421F
    classDef script fill:#FCF3CF,stroke:#F7DC6F,color:#7D6608
    classDef loss fill:#FADBD8,stroke:#EC7063,color:#641E16
```
