# rongjiehuang/generspeech architecture

> Architecture diagram of the GitHub repository rongjiehuang/generspeech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rongjiehuang/generspeech
- Repository: https://github.com/rongjiehuang/generspeech

## Overview

##### Project Type and Purpose
- This repository implements a research-grade Text-to-Speech (TTS) system (“GenerSpeech”) in PyTorch, focused on zero-shot style transfer for out-of-domain voices.
- It provides data preparation pipelines, model definitions, training scripts, and inference utilities (including a Gradio demo).

##### File Structure Analysis
Top-level directories and roles:
- assets: example audio/text for quick inference tests
- data_gen/tts: multi-step data preprocessing (text normalization, wav/text processors), forced-alignment (MFA) training, and binary data binarization
- egs: example dataset configs (LibriTTS, ESD, VCTK, LJSpeech) and scripts for data loading
- modules: core model implementations
– modules/GenerSpeech: acoustic model (generspeech.py, glow, prosody, wavenet)
– modules/fastspeech, hifigan, parallel_wavegan: alternative TTS/post-filter/vocoder modules
- inference: standalone inference scripts
– inference/GenerSpeech.py: CLI inference using acoustic model + vocoder
– inference/gradio: web UI for demo
- tasks: high-level entrypoint (tasks/run.py) for training/inference workflows via config files
- utils: shared utilities (audio I/O, checkpointing, distributed training, plotting, text encoding/norm, schedulers)
- vocoders: wrapper interfaces for Hifi-GAN, PWG

Key config & environment files:
- modules/GenerSpeech/config/generspeech.yaml – main acoustic model hyperparameters
- environment.yaml / requirements.txt – Conda/PyPI dependencies
- egs/egs_bases – reusable base YAML configs for various model components

##### README Insights
- Workflow stages clearly described: preprocess → align (MFA) → binarize → train → infer
- Pretrained checkpoints for acoustic model, vocoder, and emotion encoder
- External dependencies: Wav2Vec2 (ASR for reference feature extraction), Montreal Forced Aligner (MFA)

a. Main Components
1. Data Preparation Layer
- Raw dataset (audio + transcripts)
- Text processors, wav processors → MFA aligner → binarizer → binary dataset
2. Training Layer
- tasks/run.py orchestrates: loads binary dataset → modules/GenerSpeech acoustic model → optional parallel training (DDP) → checkpoint outputs
- Optional vocoder training (Hifi-GAN / PWG) via modules/… and tasks/vocoder
3. Inference Layer
- CLI (inference/GenerSpeech.py) & Gradio UI (inference/gradio/infer.py)
- Loads checkpoints (acoustic + vocoder + emotion encoder) + reference audio → generates mel spectrogram → vocoder → final waveform
4. External Services
- ASR model (Wav2Vec2) for reference feature extraction
- MFA for text-speech alignment

b. Relationships & Data Flows
- Inference: Reference Audio + Config → Acoustic Model → Mel → Vocoder → Output WAV
- Utilities (utils/) feed into both training and inference for logging, checkpointing, distributed setup

c. Architectural Patterns & Principles
- Modularization: clear separation of data pipelines, model definitions, training logic, inference scripts
- Config-driven design: YAML files define hyperparameters and data paths, enabling reproducibility and easy swapping of datasets/configs
- Extensibility points: modules/… folders allow plugging in new architectures (e.g., replace GenerSpeech with new model)

d. Key Technologies
- PyTorch & DistributedDataParallel (utils/ddp_utils.py)
- YAML-based configuration (Hydra-like patterns)
- MFA & Huggingface Wav2Vec2 for alignment and feature extraction
- Gradio for interactive demo

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Layer
    subgraph "Data Preparation Layer"
        direction TB
        RawData["Raw Dataset (audio + transcripts)"]:::data
        Preprocess["Preprocessing"]:::data
        click Preprocess "https://github.com/rongjiehuang/generspeech/blob/main/data_gen/tts/bin/preprocess.py"
        TextProc["Text Processors"]:::data
        click TextProc "https://github.com/rongjiehuang/generspeech/tree/main/data_gen/tts/txt_processors/"
        WavProc["Wav Processors"]:::data
        click WavProc "https://github.com/rongjiehuang/generspeech/tree/main/data_gen/tts/wav_processors/"
        MFAAlign["MFA Alignment"]:::data
        click MFAAlign "https://github.com/rongjiehuang/generspeech/blob/main/data_gen/tts/bin/train_mfa_align.py"
        Binarize["Binarization"]:::data
        click Binarize "https://github.com/rongjiehuang/generspeech/blob/main/data_gen/tts/bin/binarize.py"
        BasePre["Base Preprocess Utils"]:::data
        click BasePre "https://github.com/rongjiehuang/generspeech/blob/main/data_gen/tts/base_preprocess.py"
        BaseBin["Base Binarizers"]:::data
        click BaseBin "https://github.com/rongjiehuang/generspeech/blob/main/data_gen/tts/base_binarizer.py"
        BaseBinEmo["Base Binarizer Emotion"]:::data
        click BaseBinEmo "https://github.com/rongjiehuang/generspeech/blob/main/data_gen/tts/base_binarizer_emotion.py"
        EmotionDP["Emotion Data Pipeline"]:::data
        click EmotionDP "https://github.com/rongjiehuang/generspeech/tree/main/data_gen/tts/emotion/"
        RawData --> Preprocess
        Preprocess --> TextProc
        Preprocess --> WavProc
        TextProc --> MFAAlign
        WavProc --> MFAAlign
        MFAAlign --> Binarize
        BasePre --> Preprocess
        BaseBin --> Binarize
        BaseBinEmo --> Binarize
        EmotionDP --> Binarize
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        BinaryData["Binary Dataset (.bin)"]:::artifact
        TasksRun["Orchestrator\n tasks/run.py"]:::orchestration
        click TasksRun "https://github.com/rongjiehuang/generspeech/blob/main/tasks/run.py"
        AcousticModel["Acoustic Model\n modules/GenerSpeech"]:::model
        click AcousticModel "https://github.com/rongjiehuang/generspeech/blob/main/modules/GenerSpeech/model/generspeech.py"
        Glow["Glow Modules"]:::model
        click Glow "https://github.com/rongjiehuang/generspeech/blob/main/modules/GenerSpeech/model/glow_modules.py"
        Prosody["Prosody Utilities"]:::model
        click Prosody "https://github.com/rongjiehuang/generspeech/blob/main/modules/GenerSpeech/model/prosody_util.py"
        WaveNet["WaveNet Component"]:::model
        click WaveNet "https://github.com/rongjiehuang/generspeech/blob/main/modules/GenerSpeech/model/wavenet.py"
        FS2["FastSpeech2 (modules/fastspeech/)"]:::model
        click FS2 "https://github.com/rongjiehuang/generspeech/tree/main/modules/fastspeech/"
        VocoderTrain["Vocoder Training Tasks"]:::orchestration
        click VocoderTrain "https://github.com/rongjiehuang/generspeech/tree/main/tasks/vocoder/"
        Checkpoints["Model Checkpoints (.pt)"]:::artifact

        Binarize --> BinaryData
        BinaryData --> TasksRun
        TasksRun --> AcousticModel
        AcousticModel --> Glow
        AcousticModel --> Prosody
        AcousticModel --> WaveNet
        TasksRun --> FS2
        TasksRun --> VocoderTrain
        VocoderTrain --> VocoderTrain
        AcousticModel --> Checkpoints
        FS2 --> Checkpoints
        VocoderTrain --> Checkpoints
    end

    %% Inference Layer
    subgraph "Inference Layer"
        direction TB
        CLI["CLI Inference\n inference/GenerSpeech.py"]:::orchestration
        click CLI "https://github.com/rongjiehuang/generspeech/blob/main/inference/GenerSpeech.py"
        Gradio["Gradio Demo\n inference/gradio/infer.py"]:::orchestration
        click Gradio "https://github.com/rongjiehuang/generspeech/blob/main/inference/gradio/infer.py"
        BaseInfer["Base TTS Inference"]:::orchestration
        click BaseInfer "https://github.com/rongjiehuang/generspeech/blob/main/inference/base_tts_infer.py"
        VocoderWrap["Vocoder Wrapper Interfaces\n vocoders/"]:::model
        click VocoderWrap "https://github.com/rongjiehuang/generspeech/tree/main/vocoders/"
        OutputWav["Output WAV"]:::artifact

        Checkpoints --> CLI
        Checkpoints --> Gradio
        CLI --> BaseInfer
        Gradio --> BaseInfer
        BaseInfer --> AcousticModel
        AcousticModel -->|"mel spectrogram"| VocoderWrap
        VocoderWrap -->|"waveform"| OutputWav
    end

    %% External Services & Configs
    subgraph "External & Config"
        direction TB
        ASR["Wav2Vec2 ASR Model"]:::external
        MFAService["Montreal Forced Aligner"]:::external
        ConfigModel["Model Config\ngenerspeech.yaml"]:::config
        click ConfigModel "https://github.com/rongjiehuang/generspeech/blob/main/modules/GenerSpeech/config/generspeech.yaml"
        ConfigBase["Base YAML Configs\negs/egs_bases/tts/"]:::config
        click ConfigBase "https://github.com/rongjiehuang/generspeech/tree/main/egs/egs_bases/tts/"
        Utils["Shared Utilities\n utils/"]:::util
        click Utils "https://github.com/rongjiehuang/generspeech/tree/main/utils/"

        ASR --> BaseInfer
        MFAService --> MFAAlign
        ConfigModel --> TasksRun
        ConfigModel --> BaseInfer
        ConfigBase --> TasksRun
        Utils --> TasksRun
        Utils --> BaseInfer
    end

    %% Styles
    classDef data fill:#D6EAF8,stroke:#2E86C1;
    classDef model fill:#D5F5E3,stroke:#28B463;
    classDef orchestration fill:#FCF3CF,stroke:#F1C40F;
    classDef external fill:#F2F3F4,stroke:#7F8C8D;
    classDef config fill:#EDBB99,stroke:#D35400;
    classDef util fill:#E8DAEF,stroke:#8E44AD;
    classDef artifact fill:#FDEDEC,stroke:#E74C3C;
```
