# ming024/fastspeech2 architecture

> Architecture diagram of the GitHub repository ming024/fastspeech2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ming024/fastspeech2
- Repository: https://github.com/ming024/fastspeech2

## Overview

##### Project type and purpose
- This is a full-pipeline, deep-learning-based text-to-speech (TTS) system in PyTorch, implementing Microsoft’s FastSpeech 2 algorithm.
- It handles the entire workflow: raw data alignment, feature extraction, model training, waveform synthesis via a neural vocoder, and inference/demo.

##### File structure
Top-level directories and files:
- config/          – YAML configs for dataset-specific preprocessing, model hyperparameters, training schedules
- preprocessor/    – Dataset-specific preprocessing wrappers (AISHELL3, LJSpeech, LibriTTS) and a general entry point
- prepare_align.py – Script to run Montreal Forced Aligner (MFA) or prepare its outputs
- preprocess.py    – Main data preparation: text/audio → phonemes/mel-spectrograms
- preprocessed_data/ – Storage for processed features, metadata, train/val splits
- text/            – Text cleaning and phoneme mapping modules
- audio/           – Signal processing utilities (STFT, tools)
- dataset.py       – DataLoader definitions assembling text + audio features
- transformer/     – Core transformer building blocks (Layers, SubLayers, Constants, Models)
- model/           – FastSpeech2 model definition, loss functions, optimizer wrapper
- train.py         – Training orchestration
- synthesize.py    – Inference orchestration (text → mel → wave)
- evaluate.py      – Quantitative evaluation scripts
- hifigan/         – HiFi-GAN neural vocoder code and pretrained checkpoints
- demo/            – Precomputed ground-truth vs. synthesized WAV samples for each dataset
- utils/           – Generic utilities (model loader, plotting, audio save)
- lexicon/         – Pronunciation lexicons for English/Mandarin
- output/          – (implicit) target for logs, checkpoints, synthesized results
- index.html, img/ – Static assets for the README site

##### Extract architecture from README
- Shows a three-stage pipeline:
a. Preprocessing & Alignment (with MFA)
b. FastSpeech 2 acoustic model (text → mel)
c. Neural vocoder (MelGAN/HiFi-GAN) for waveform synthesis
- Supports multi-speaker and multi-language via speaker IDs and dataset-specific configs
- Demonstrates control over duration, energy, pitch via inference flags

##### Main components & interactions
a. External Service:
- Montreal Forced Aligner (MFA) – consumes raw audio + transcripts, emits time-aligned TextGrids
b. Preprocessing Pipeline:
- prepare_align.py → runs MFA or extracts TextGrids
- preprocess.py + preprocessor/* → text cleaning (text/), feature extraction (audio/): mel-spectrograms, pitch, energy
- Outputs: preprocessed_data/[Dataset]/{mel-bins, durations, pitch, energy, metadata, train.txt, val.txt}
c. Data Loader:
- dataset.py – reads preprocessed_data, lexicon/*, builds PyTorch Dataset & DataLoader
d. Acoustic Model – FastSpeech 2:
- transformer/* – attention blocks, positional encodings
- model/fastspeech2.py – encoder, variance adaptors, decoder, post-net
- model/loss.py & optimizer.py – loss calculation, scheduler, optimizer
e. Training Orchestration:
- train.py – ties DataLoader + model + loss, logs to output/log/[Dataset] via TensorBoard
f. Inference Orchestration:
- synthesize.py – loads checkpoint, takes raw text + speaker_id, runs text→mel via model, applies duration/energy/pitch control
g. Neural Vocoder:
- hifigan/models.py & generator_*.pth.tar.zip – convert mel to waveform (optional MelGAN code could be similarly added)
h. Output:
- synthesize.py writes WAV to output/result/, demo/ provides sample comparisons
- evaluate.py reads predictions + ground truth to compute objective metrics

##### Architectural patterns & design principles
- Modular pipeline: clear separation of concerns (alignment, preprocessing, modeling, vocoding)
- Config-driven: all datasets and hyperparameters externalized into YAML under config/
- Reusable components: transformer blocks shared between training and inference
- Extensibility: add new datasets by dropping YAML + a preprocessor file
- Data flow in one direction: raw data → features → model → output; training and inference share subcomponents

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Preparation Stage
    subgraph "Data Preparation" 
        direction TB
        RawAudio["Raw Audio + Transcripts"]:::external
        MFAService(["MFA Service"]):::external
        PrepareAlign["prepare_align.py"]:::process
        PreprocessEntry["preprocess.py"]:::process
        subgraph "Preprocessor Modules"
            direction TB
            PreprocCore["preprocessor/preprocessor.py"]:::process
            AIShell3Prep["preprocessor/aishell3.py"]:::process
            LJSpeechPrep["preprocessor/ljspeech.py"]:::process
            LibriTTSPrep["preprocessor/libritts.py"]:::process
        end
        subgraph "Text Cleaning Modules"
            direction TB
            Cleaners["text/cleaners.py"]:::process
            CMUDict["text/cmudict.py"]:::process
            Numbers["text/numbers.py"]:::process
            Pinyin["text/pinyin.py"]:::process
            Symbols["text/symbols.py"]:::process
        end
        subgraph "Audio Feature Utils"
            direction TB
            AudioProc["audio/audio_processing.py"]:::process
            STFT["audio/stft.py"]:::process
            AudioTools["audio/tools.py"]:::process
        end
        PreprocessedStorage["preprocessed_data/"]:::store
        ConfigFiles["config/"]:::store
    end

    RawAudio -->|audio+txt| MFAService
    MFAService -->|TextGrid| PrepareAlign
    PrepareAlign -->|aligned data| PreprocessEntry
    PreprocessEntry --> PreprocCore
    PreprocCore --> AIShell3Prep
    PreprocCore --> LJSpeechPrep
    PreprocCore --> LibriTTSPrep
    PreprocessEntry --> Cleaners
    PreprocessEntry --> CMUDict
    PreprocessEntry --> Numbers
    PreprocessEntry --> Pinyin
    PreprocessEntry --> Symbols
    PreprocessEntry --> AudioProc
    PreprocessEntry --> STFT
    PreprocessEntry --> AudioTools
    PreprocessEntry -->|features| PreprocessedStorage
    ConfigFiles --> PrepareAlign
    ConfigFiles --> PreprocessEntry

    %% Model Lifecycle Stage
    subgraph "Model Lifecycle"
        direction TB
        DatasetLoader["dataset.py"]:::process
        Lexicons["lexicon/"]:::store
        subgraph "Acoustic Model"
            direction TB
            subgraph "Transformer Blocks"
                Constants["transformer/Constants.py"]:::process
                Layers["transformer/Layers.py"]:::process
                SubLayers["transformer/SubLayers.py"]:::process
                Modules["transformer/Modules.py"]:::process
                ModelsTF["transformer/Models.py"]:::process
            end
            FastSpeech2["model/fastspeech2.py"]:::process
            Losses["model/loss.py"]:::process
            Optimizer["model/optimizer.py"]:::process
        end
        TrainController["train.py"]:::process
        Checkpoints["output/ckpt/"]:::store
        Logs["output/log/"]:::store
    end

    PreprocessedStorage -->|data| DatasetLoader
    Lexicons --> DatasetLoader
    ConfigFiles --> DatasetLoader
    DatasetLoader --> FastSpeech2
    Constants --> FastSpeech2
    Layers --> FastSpeech2
    SubLayers --> FastSpeech2
    Modules --> FastSpeech2
    ModelsTF --> FastSpeech2
    FastSpeech2 --> Losses
    Losses --> Optimizer
    Optimizer --> TrainController
    FastSpeech2 --> TrainController
    DatasetLoader --> TrainController
    TrainController -->|checkpoint| Checkpoints
    TrainController -->|logs| Logs

    %% Inference & Synthesis Stage
    subgraph "Inference & Synthesis"
        direction TB
        Synthesize["synthesize.py"]:::process
        Evaluate["evaluate.py"]:::process
        subgraph "Neural Vocoder"
            direction TB
            HiFiGANModel["hifigan/models.py"]:::process
            HiFiCheckpoint1["hifigan/generator_LJSpeech.pth.tar.zip"]:::store
            HiFiCheckpoint2["hifigan/generator_universal.pth.tar.zip"]:::store
            HiFiConfig["hifigan/config.json"]:::store
        end
        ResultWAV["output/result/"]:::store
        DemoWAV["demo/"]:::store
    end

    Checkpoints --> Synthesize
    Synthesize --> FastSpeech2
    Synthesize -->|mel-spectrogram| HiFiGANModel
    HiFiGANModel -->|waveform| ResultWAV
    Synthesize --> DemoWAV
    Evaluate -->|metrics| Logs
    Evaluate --> Checkpoints

    ConfigFiles --> Synthesize
    ConfigFiles --> Evaluate

    %% Utility & Assets
    subgraph "Utilities & Assets"
        direction TB
        UtilsModel["utils/model.py"]:::process
        UtilsTools["utils/tools.py"]:::process
        IndexHTML["index.html"]:::store
        ImageAssets["img/"]:::store
    end

    UtilsModel --> TrainController
    UtilsModel --> Synthesize
    UtilsTools --> PreprocessEntry
    UtilsTools --> TrainController
    UtilsTools --> Evaluate

    %% Click Events
    click PrepareAlign "https://github.com/ming024/fastspeech2/blob/master/prepare_align.py"
    click ConfigFiles "https://github.com/ming024/fastspeech2/tree/master/config/"
    click PreprocessEntry "https://github.com/ming024/fastspeech2/blob/master/preprocess.py"
    click PreprocCore "https://github.com/ming024/fastspeech2/blob/master/preprocessor/preprocessor.py"
    click AIShell3Prep "https://github.com/ming024/fastspeech2/blob/master/preprocessor/aishell3.py"
    click LJSpeechPrep "https://github.com/ming024/fastspeech2/blob/master/preprocessor/ljspeech.py"
    click LibriTTSPrep "https://github.com/ming024/fastspeech2/blob/master/preprocessor/libritts.py"
    click Cleaners "https://github.com/ming024/fastspeech2/blob/master/text/cleaners.py"
    click CMUDict "https://github.com/ming024/fastspeech2/blob/master/text/cmudict.py"
    click Numbers "https://github.com/ming024/fastspeech2/blob/master/text/numbers.py"
    click Pinyin "https://github.com/ming024/fastspeech2/blob/master/text/pinyin.py"
    click Symbols "https://github.com/ming024/fastspeech2/blob/master/text/symbols.py"
    click AudioProc "https://github.com/ming024/fastspeech2/blob/master/audio/audio_processing.py"
    click STFT "https://github.com/ming024/fastspeech2/blob/master/audio/stft.py"
    click AudioTools "https://github.com/ming024/fastspeech2/blob/master/audio/tools.py"
    click PreprocessedStorage "https://github.com/ming024/fastspeech2/tree/master/preprocessed_data/"
    click DatasetLoader "https://github.com/ming024/fastspeech2/blob/master/dataset.py"
    click Lexicons "https://github.com/ming024/fastspeech2/tree/master/lexicon/"
    click Constants "https://github.com/ming024/fastspeech2/blob/master/transformer/Constants.py"
    click Layers "https://github.com/ming024/fastspeech2/blob/master/transformer/Layers.py"
    click SubLayers "https://github.com/ming024/fastspeech2/blob/master/transformer/SubLayers.py"
    click Modules "https://github.com/ming024/fastspeech2/blob/master/transformer/Modules.py"
    click ModelsTF "https://github.com/ming024/fastspeech2/blob/master/transformer/Models.py"
    click FastSpeech2 "https://github.com/ming024/fastspeech2/blob/master/model/fastspeech2.py"
    click Losses "https://github.com/ming024/fastspeech2/blob/master/model/loss.py"
    click Optimizer "https://github.com/ming024/fastspeech2/blob/master/model/optimizer.py"
    click TrainController "https://github.com/ming024/fastspeech2/blob/master/train.py"
    click Synthesize "https://github.com/ming024/fastspeech2/blob/master/synthesize.py"
    click Evaluate "https://github.com/ming024/fastspeech2/blob/master/evaluate.py"
    click HiFiGANModel "https://github.com/ming024/fastspeech2/blob/master/hifigan/models.py"
    click HiFiCheckpoint1 "https://github.com/ming024/fastspeech2/blob/master/hifigan/generator_LJSpeech.pth.tar.zip"
    click HiFiCheckpoint2 "https://github.com/ming024/fastspeech2/blob/master/hifigan/generator_universal.pth.tar.zip"
    click HiFiConfig "https://github.com/ming024/fastspeech2/blob/master/hifigan/config.json"
    click DemoWAV "https://github.com/ming024/fastspeech2/tree/master/demo/"
    click UtilsModel "https://github.com/ming024/fastspeech2/blob/master/utils/model.py"
    click UtilsTools "https://github.com/ming024/fastspeech2/blob/master/utils/tools.py"
    click IndexHTML "https://github.com/ming024/fastspeech2/blob/master/index.html"
    click ImageAssets "https://github.com/ming024/fastspeech2/tree/master/img/"

    %% Styles
    classDef process fill:#B3CDE3,stroke:#4B8BBE,color:#000
    classDef store fill:#CCEBC5,stroke:#33A02C,color:#000
    classDef external fill:#D9D9D9,stroke:#999,color:#000
```
