How xcmyz/fastspeech works
1. This repository is a PyTorch implementation of FastSpeech for text-to-speech. The architecture is split into three main planes: data preprocessing, the FastSpeech acoustic model, and waveform generation for evaluation/sample synthesis.
2. The input domain is LJSpeech. Raw audio and transcripts live under data/, while alignments.zip provides the phoneme-to-frame duration targets needed by FastSpeech. Those alignments are a key boundary: they replace end-to-end attention training with supervised duration learning.
3. The text front-end is in text/. It handles normalization and symbol conversion through cleaners, number expansion, CMU dictionary lookups, and symbol definitions. This subsystem turns transcripts into model-ready token sequences.
4. The audio front-end is in audio/. It provides STFT, mel-spectrogram, and waveform utilities plus audio-specific hyperparameters. This is the boundary between raw .wav files and the acoustic features the model trains on.
5. preprocess.py is the offline feature-building stage. It combines LJSpeech audio, cleaned text, and alignment information to generate the training artifacts consumed later by the dataset and model code. This is where the repo creates the supervised training pairs for FastSpeech.
6. dataset.py and data/ljspeech.py define how those precomputed artifacts are loaded during training. They bridge the stored features with batched PyTorch tensors, including text sequences, mel targets, and duration information.
7. The core FastSpeech model is centered in model.py, with supporting building blocks in modules.py and the vendored transformer/ package. The transformer directory provides the encoder/decoder-style layers and attention/feed-forward primitives used by the architecture.
8. loss.py and optimizer.py define the training objective and optimization behavior. The loss side is where the model is supervised against mel-spectrogram targets and duration targets; the optimizer side controls how the acoustic model is updated during train.py.
9. train.py is the main training entry point. It ties together preprocessing outputs, the dataset loader, the FastSpeech model, loss computation, optimization, checkpointing, and periodic logging.
10. eval.py is the inference/evaluation entry point. It loads a trained FastSpeech checkpoint, converts text to predicted mel spectrograms, and then hands those mels off to a vocoder for audio rendering.
11. Waveform synthesis is handled by the separate waveglow/ package. This repo does not generate raw audio directly from FastSpeech; instead it uses a pretrained NVIDIA WaveGlow checkpoint as an external vocoder boundary to convert predicted mel features into .wav files.
12. sample/ contains generated example outputs, showing the end product of the inference pipeline. img/ is documentation-only, while requirements.txt captures the PyTorch/audio stack needed to run the full pipeline.
13. Overall data flow: LJSpeech + alignments -> preprocessing -> stored text/mel/duration features -> dataset loader -> FastSpeech training -> checkpoint -> evaluation/inference -> mel spectrograms -> WaveGlow vocoder -> audio samples.
Main entry points: model.py FastSpeech · train.py Train · preprocess.py Preprocess · dataset.py Dataset
Components 17
Data Assets
Raw corpus, alignments, and example outputs.
- LJSpeech data · Raw audio and transcripts used as the training source.
data - Alignments · Phoneme-to-frame durations used for supervised FastSpeech training.
alignments.zip - Samples · Example synthesized waveforms produced by the inference pipeline.
sample
Front-Ends
Text and audio conversion layers.
Preprocessing
Offline feature generation and dataset assembly.
- Preprocess · Builds supervised training artifacts from audio, text, and alignments.
preprocess.py - Dataset · Loads precomputed text, mel, and duration tensors for training.
dataset.py - LJSpeech IO · Dataset-specific helpers for reading LJSpeech-derived artifacts.
data/ljspeech.py
Acoustic Model
FastSpeech network and supporting transformer internals.
- FastSpeech · Core duration-based text-to-mel synthesis model.
model.py - Modules · Reusable building blocks used by the FastSpeech architecture.
modules.py - Transformer · Vendored attention, feed-forward, and layer primitives.
transformer
Training
Losses, optimization, and runtime entry points.
Vocoder
Mel-to-waveform synthesis boundary.
- WaveGlow · Mel-to-waveform synthesis package used as the audio rendering boundary.
waveglow - Vocoder bridge · Glue code that converts predicted mels into audio with WaveGlow.
glow.py
Connections 19
- LJSpeech data → Preprocess: raw audio/text · Source corpus is consumed during offline preprocessing.
- Alignments → Preprocess: durations · Alignment targets are converted into supervised duration data.
- Text → Preprocess: clean text · Normalized symbol sequences are built from transcripts.
- Audio → Preprocess: mel features · Audio utilities produce mel/STFT features for training artifacts.
- Preprocess → Dataset: stored features · Offline outputs are loaded back as training examples.
- Dataset → LJSpeech IO: LJSpeech batches · Dataset-specific access shapes batch inputs and targets.
- Dataset → Train: mini-batches · Batched tensors feed the training loop.
- LJSpeech IO → Train: dataset access · Dataset helpers support the training pipeline.
- Text → FastSpeech: token ids · Text normalization feeds tokenized inputs to the model.
- FastSpeech → Modules: uses · Core architecture is assembled from reusable blocks.
- FastSpeech → Transformer: layers · Transformer primitives provide attention and feed-forward internals.
- FastSpeech → Loss: predictions · Model outputs are compared against mel and duration targets.
- Loss → Optimizer: gradients · Training objective drives optimization updates.
- Optimizer → Train: step updates · Optimizer config is applied inside the training loop.
- Train → FastSpeech: fit · Training updates the acoustic model parameters.
- Train → Eval: checkpoint · Trained checkpoints are consumed by evaluation.
- Eval → Vocoder bridge: mel spectrograms · Evaluation emits predicted mels for waveform rendering.
- Vocoder bridge → WaveGlow: synthesize · Bridge calls the WaveGlow vocoder package.
- WaveGlow → Samples: wav output · Generated example audio is written as sample files.