# brian-mwangi-developer/styletts2-prep architecture

> Architecture diagram of the GitHub repository brian-mwangi-developer/styletts2-prep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brian-mwangi-developer/styletts2-prep
- Repository: https://github.com/brian-mwangi-developer/styletts2-prep

## Overview

##### Project type and purpose
- This is a research-grade, full-stack machine-learning pipeline for text-to-speech (TTS) synthesis (“StyleTTS2”).
- It implements a two-stage training pipeline: (1) latent-style diffusion TTS network and (2) adversarial fine-tuning with large speech-language model (SLM) discriminators.
- It supports data preprocessing (text alignment, pitch extraction, PL-BERT text embedding), model training, inference notebooks, a simple HTTP API and Docker deployment.

##### High-level components and directories
- Data layer (folder “Data”): raw .wav files, train/val lists, OOD texts.
- Config management (“Configs”): YAML configs for each dataset and finetuning.
- Preprocessing utilities (“Utils/ASR”, “Utils/JDC”, “Utils/PLBERT”, “text_utils.py”): text aligner, pitch extractor, BERT embeddings, phonemizer.
- Core model modules (“Modules”):
– diffusion/…: latent diffusion sampler, denoiser, utilities
– discriminators.py (SLM adversarial losses)
– hifigan.py (vocoder), istftnet.py (inverse STFT vocoder)
– slmadv.py (adversarial wrapper)
– utils.py (common functions)
- Training orchestration (root):
– train_first.py (stage-1 training via Accelerate)
– train_second.py (stage-2 adversarial fine-tuning, DP fallback)
– train_finetune(.py, _accelerate.py): finetuning on new speakers
- Inference and API:
– libri_inference.py, api.py (lightweight HTTP inference server)
– Demo notebooks (Colab + Jupyter in “Demo”)
– Dockerfile.api (serving), Dockerfile.train (training environment)
- Models (“Models”): checkpoints, sample outputs, TensorBoard logs

##### Relationships & data flow
1. Raw data → Preprocessor:
- Text aligner (ASR) → phoneme durations
- Pitch extractor (JDC) → F0 contour
- PL-BERT (text embedding) → contextual text features
2. Stage-1 TTS network:
- Text embeddings + durations + F0 → diffusion-based style sampler → mel-spectrogram decoder
- Losses: reconstruction (L1/L2), alignment, diffusion denoising
3. Stage-2 adversarial training:
- Generated mel → vocoder (HiFiGAN or iSTFTNet) → waveform
- Discriminators: large pre-trained SLMs (WavLM) + HifiGAN critics → adversarial loss backprop
4. Inference pipeline:
Text input → phonemizer → PL-BERT → style diffusion → mel decoder → vocoder → audio → API endpoint/notebook

##### Architectural patterns & principles
- Modular layers separating preprocessing, model definition, training scripts, inference/API.
- Two-stage pipeline: diffusion generator + adversarial fine-tuning (sequential stages).
- Pluggable pretrained modules (ASR, JDC, PL-BERT) loaded as external dependencies.
- Config-driven orchestration for datasets, hyperparameters, multi- vs. single-speaker mode.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Data["Data Layer\n(raw .wav, train/val lists)"]:::data
    end

    %% Config Management
    subgraph "Config Management"
        direction TB
        Configs["Configs\n(YAML configs)"]:::config
    end

    %% Preprocessing Modules
    subgraph "Preprocessing Modules"
        direction TB
        ASR["ASR Preprocessor\n(text aligner)"]:::pre
        JDC["JDC Preprocessor\n(pitch extractor)"]:::pre
        PLBERT["PL-BERT Preprocessor\n(text embedding)"]:::pre
        TextUtils["Text Utilities\n(phonemizer)"]:::pre
    end

    %% Stage-1 TTS Network
    subgraph "Stage-1 TTS Network"
        direction TB
        Diffusion["Diffusion Network\n(latent diffusion)"]:::stage1
        UtilsMod["Core Utilities\n(Modules/utils.py)"]:::stage1
    end

    %% Stage-2 Vocoder & Discriminators
    subgraph "Stage-2 Vocoder & Discriminators"
        direction TB
        Wrapper["Adversarial Wrapper\n(slmadv.py)"]:::stage2
        Discriminators["SLM Discriminators\n(discriminators.py)"]:::stage2
        VocoderHiFiGAN["HiFiGAN Vocoder\n(hifigan.py)"]:::stage2
        VocoderISTFT["iSTFTNet Vocoder\n(istftnet.py)"]:::stage2
    end

    %% Training Orchestrator
    subgraph "Training Orchestrator"
        direction TB
        Train1["Stage-1 Training\n(train_first.py)"]:::orch
        Train2["Stage-2 Training\n(train_second.py)"]:::orch
        Finetune["Speaker Finetuning\n(train_finetune*.py)"]:::orch
        CommonTrain["Common Utils\n(meldataset.py, losses.py, optimizers.py)"]:::orch
        Req["Requirements\n(requirements.txt)"]:::orch
    end

    %% Inference & Deployment
    subgraph "Inference & Deployment"
        direction TB
        API["HTTP API\n(api.py)"]:::infra
        LibriInf["Libri Inference\n(libri_inference.py)"]:::infra
        Notebooks["Demo Notebooks\n(Demo/)"]:::infra
        DockerAPI["Docker API\n(Dockerfile.api)"]:::infra
        DockerTrain["Docker Train\n(Dockerfile.train)"]:::infra
        Models["Saved Models & Logs\n(Models/)"]:::infra
    end

    %% Data Flow
    Data -->|"raw audio"| ASR
    Data -->|"raw audio"| JDC
    Data -->|"text"| PLBERT
    Data -->|"text"| TextUtils

    ASR -->|"phoneme durations"| Diffusion
    JDC -->|"F0 contours"| Diffusion
    PLBERT -->|"text embeddings"| Diffusion
    TextUtils -->|"phonemes"| Diffusion

    Configs --> Train1
    Configs --> Train2
    Configs --> Finetune

    Train1 --> Diffusion
    Diffusion -->|"mel-spectrogram"| Wrapper
    Wrapper --> Discriminators
    Wrapper --> VocoderHiFiGAN
    Wrapper --> VocoderISTFT
    Discriminators -->|"adversarial loss"| Train2

    VocoderHiFiGAN -->|"waveform"| API
    VocoderISTFT -->|"waveform"| API
    API --> Notebooks
    LibriInf --> API

    Models --> API
    Models --> Notebooks

    CommonTrain --> Train1
    CommonTrain --> Train2
    Req --> Train1
    Req --> Train2

    %% Click Events
    click Data "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Data/"
    click Configs "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Configs/"
    click ASR "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Utils/ASR/"
    click JDC "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Utils/JDC/"
    click PLBERT "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Utils/PLBERT/"
    click TextUtils "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/text_utils.py"
    click Diffusion "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Modules/diffusion/"
    click Discriminators "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Modules/discriminators.py"
    click VocoderHiFiGAN "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Modules/hifigan.py"
    click VocoderISTFT "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Modules/istftnet.py"
    click Wrapper "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Modules/slmadv.py"
    click UtilsMod "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Modules/utils.py"
    click Train1 "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/train_first.py"
    click Train2 "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/train_second.py"
    click Finetune "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/train_finetune.py"
    click CommonTrain "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/meldataset.py"
    click Req "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/requirements.txt"
    click API "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/api.py"
    click LibriInf "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/libri_inference.py"
    click Notebooks "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Demo/"
    click DockerAPI "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Dockerfile.api"
    click DockerTrain "https://github.com/brian-mwangi-developer/styletts2-prep/blob/main/Dockerfile.train"
    click Models "https://github.com/brian-mwangi-developer/styletts2-prep/tree/main/Models/"

    %% Styles
    classDef data fill:#B3DDF2,stroke:#006D9C;
    classDef config fill:#D2B4DE,stroke:#7D3C98;
    classDef pre fill:#AED6F1,stroke:#21618C;
    classDef stage1 fill:#A9DFBF,stroke:#27AE60;
    classDef stage2 fill:#76D7C4,stroke:#148F77;
    classDef orch fill:#F9E79F,stroke:#B7950B;
    classDef infra fill:#F5B7B1,stroke:#C0392B;
```
