How ming024/fastspeech2 works
Project type and purpose
• This is a full-pipeline, deep-learning-based text-to-speech (TTS) system in PyTorch, implementing Microsoft’s FastSpeech 2 algorithm.
• It handles the entire workflow: raw data alignment, feature extraction, model training, waveform synthesis via a neural vocoder, and inference/demo.
File structure
Top-level directories and files:
• config/ – YAML configs for dataset-specific preprocessing, model hyperparameters, training schedules
• preprocessor/ – Dataset-specific preprocessing wrappers (AISHELL3, LJSpeech, LibriTTS) and a general entry point
• prepare_align.py – Script to run Montreal Forced Aligner (MFA) or prepare its outputs
• preprocess.py – Main data preparation: text/audio → phonemes/mel-spectrograms
• preprocessed_data/ – Storage for processed features, metadata, train/val splits
• text/ – Text cleaning and phoneme mapping modules
• audio/ – Signal processing utilities (STFT, tools)
• dataset.py – DataLoader definitions assembling text + audio features
• transformer/ – Core transformer building blocks (Layers, SubLayers, Constants, Models)
• model/ – FastSpeech2 model definition, loss functions, optimizer wrapper
• train.py – Training orchestration
• synthesize.py – Inference orchestration (text → mel → wave)
• evaluate.py – Quantitative evaluation scripts
• hifigan/ – HiFi-GAN neural vocoder code and pretrained checkpoints
• demo/ – Precomputed ground-truth vs. synthesized WAV samples for each dataset
• utils/ – Generic utilities (model loader, plotting, audio save)
• lexicon/ – Pronunciation lexicons for English/Mandarin
• output/ – (implicit) target for logs, checkpoints, synthesized results
• index.html, img/ – Static assets for the README site
Extract architecture from README
• Shows a three-stage pipeline:
a. Preprocessing & Alignment (with MFA)
b. FastSpeech 2 acoustic model (text → mel)
c. Neural vocoder (MelGAN/HiFi-GAN) for waveform synthesis
• Supports multi-speaker and multi-language via speaker IDs and dataset-specific configs
• Demonstrates control over duration, energy, pitch via inference flags
Main components & interactions
a. External Service:
• Montreal Forced Aligner (MFA) – consumes raw audio + transcripts, emits time-aligned TextGrids
b. Preprocessing Pipeline:
• prepare_align.py → runs MFA or extracts TextGrids
• preprocess.py + preprocessor/* → text cleaning (text/), feature extraction (audio/): mel-spectrograms, pitch, energy
• Outputs: preprocessed_data/[Dataset]/{mel-bins, durations, pitch, energy, metadata, train.txt, val.txt}
c. Data Loader:
• dataset.py – reads preprocessed_data, lexicon/*, builds PyTorch Dataset & DataLoader
d. Acoustic Model – FastSpeech 2:
• transformer/* – attention blocks, positional encodings
• model/fastspeech2.py – encoder, variance adaptors, decoder, post-net
• model/loss.py & optimizer.py – loss calculation, scheduler, optimizer
e. Training Orchestration:
• train.py – ties DataLoader + model + loss, logs to output/log/[Dataset] via TensorBoard
f. Inference Orchestration:
• synthesize.py – loads checkpoint, takes raw text + speaker_id, runs text→mel via model, applies duration/energy/pitch control
g. Neural Vocoder:
• hifigan/models.py & generator_*.pth.tar.zip – convert mel to waveform (optional MelGAN code could be similarly added)
h. Output:
• synthesize.py writes WAV to output/result/, demo/ provides sample comparisons
• evaluate.py reads predictions + ground truth to compute objective metrics
Architectural patterns & design principles
• Modular pipeline: clear separation of concerns (alignment, preprocessing, modeling, vocoding)
• Config-driven: all datasets and hyperparameters externalized into YAML under config/
• Reusable components: transformer blocks shared between training and inference
• Extensibility: add new datasets by dropping YAML + a preprocessor file
• Data flow in one direction: raw data → features → model → output; training and inference share subcomponents