# xcmyz/fastspeech architecture

> Architecture diagram of the GitHub repository xcmyz/fastspeech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xcmyz/fastspeech
- Repository: https://github.com/xcmyz/fastspeech

## Overview

1. This repository is a PyTorch implementation of FastSpeech for text-to-speech. The architecture is split into three main planes: data preprocessing, the FastSpeech acoustic model, and waveform generation for evaluation/sample synthesis.

2. The input domain is LJSpeech. Raw audio and transcripts live under `data/`, while `alignments.zip` provides the phoneme-to-frame duration targets needed by FastSpeech. Those alignments are a key boundary: they replace end-to-end attention training with supervised duration learning.

3. The text front-end is in `text/`. It handles normalization and symbol conversion through cleaners, number expansion, CMU dictionary lookups, and symbol definitions. This subsystem turns transcripts into model-ready token sequences.

4. The audio front-end is in `audio/`. It provides STFT, mel-spectrogram, and waveform utilities plus audio-specific hyperparameters. This is the boundary between raw `.wav` files and the acoustic features the model trains on.

5. `preprocess.py` is the offline feature-building stage. It combines LJSpeech audio, cleaned text, and alignment information to generate the training artifacts consumed later by the dataset and model code. This is where the repo creates the supervised training pairs for FastSpeech.

6. `dataset.py` and `data/ljspeech.py` define how those precomputed artifacts are loaded during training. They bridge the stored features with batched PyTorch tensors, including text sequences, mel targets, and duration information.

7. The core FastSpeech model is centered in `model.py`, with supporting building blocks in `modules.py` and the vendored `transformer/` package. The transformer directory provides the encoder/decoder-style layers and attention/feed-forward primitives used by the architecture.

8. `loss.py` and `optimizer.py` define the training objective and optimization behavior. The loss side is where the model is supervised against mel-spectrogram targets and duration targets; the optimizer side controls how the acoustic model is updated during `train.py`.

9. `train.py` is the main training entry point. It ties together preprocessing outputs, the dataset loader, the FastSpeech model, loss computation, optimization, checkpointing, and periodic logging.

10. `eval.py` is the inference/evaluation entry point. It loads a trained FastSpeech checkpoint, converts text to predicted mel spectrograms, and then hands those mels off to a vocoder for audio rendering.

11. Waveform synthesis is handled by the separate `waveglow/` package. This repo does not generate raw audio directly from FastSpeech; instead it uses a pretrained NVIDIA WaveGlow checkpoint as an external vocoder boundary to convert predicted mel features into `.wav` files.

12. `sample/` contains generated example outputs, showing the end product of the inference pipeline. `img/` is documentation-only, while `requirements.txt` captures the PyTorch/audio stack needed to run the full pipeline.

13. Overall data flow: LJSpeech + alignments -> preprocessing -> stored text/mel/duration features -> dataset loader -> FastSpeech training -> checkpoint -> evaluation/inference -> mel spectrograms -> WaveGlow vocoder -> audio samples.

Main entry points: [`model.py`](https://github.com/xcmyz/fastspeech/blob/master/model.py) (FastSpeech), [`train.py`](https://github.com/xcmyz/fastspeech/blob/master/train.py) (Train), [`preprocess.py`](https://github.com/xcmyz/fastspeech/blob/master/preprocess.py) (Preprocess), [`dataset.py`](https://github.com/xcmyz/fastspeech/blob/master/dataset.py) (Dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_assets["Data Assets"]
  node_ljspeech_data[("LJSpeech data<br/>raw corpus")]
  node_alignments_zip["Alignments<br/>duration targets<br/>[alignments.zip]"]
  node_sample_outputs[("Samples<br/>generated audio")]
end

subgraph group_frontends["Front-Ends"]
  node_text_frontend["Text<br/>normalization"]
  node_audio_frontend["Audio<br/>feature utils"]
end

subgraph group_preprocessing["Preprocessing"]
  node_preprocess_stage["Preprocess<br/>offline build<br/>[preprocess.py]"]
  node_dataset_loader["Dataset<br/>batch loader<br/>[dataset.py]"]
  node_ljspeech_reader["LJSpeech IO<br/>dataset bridge<br/>[ljspeech.py]"]
end

subgraph group_acoustic_model["Acoustic Model"]
  node_fastspeech_model["FastSpeech<br/>acoustic model<br/>[model.py]"]
  node_model_modules["Modules<br/>model blocks<br/>[modules.py]"]
  node_transformer_pkg["Transformer<br/>layers package"]
end

subgraph group_training["Training"]
  node_losses["Loss<br/>training objective<br/>[loss.py]"]
  node_optimizer_cfg["Optimizer<br/>update rule<br/>[optimizer.py]"]
  node_train_entry["Train<br/>entry point<br/>[train.py]"]
  node_eval_entry["Eval<br/>entry point<br/>[eval.py]"]
end

subgraph group_vocoder["Vocoder"]
  node_waveglow_pkg["WaveGlow<br/>vocoder package"]
  node_waveglow_bridge["Vocoder bridge<br/>inference helper<br/>[glow.py]"]
end

node_ljspeech_data -->|"raw audio/text"| node_preprocess_stage
node_alignments_zip -->|"durations"| node_preprocess_stage
node_text_frontend -->|"clean text"| node_preprocess_stage
node_audio_frontend -->|"mel features"| node_preprocess_stage
node_preprocess_stage -->|"stored features"| node_dataset_loader
node_dataset_loader -->|"LJSpeech batches"| node_ljspeech_reader
node_dataset_loader -->|"mini-batches"| node_train_entry
node_ljspeech_reader -->|"dataset access"| node_train_entry
node_text_frontend -->|"token ids"| node_fastspeech_model
node_fastspeech_model -->|"uses"| node_model_modules
node_fastspeech_model -->|"layers"| node_transformer_pkg
node_fastspeech_model -->|"predictions"| node_losses
node_losses -->|"gradients"| node_optimizer_cfg
node_optimizer_cfg -->|"step updates"| node_train_entry
node_train_entry -->|"fit"| node_fastspeech_model
node_train_entry -->|"checkpoint"| node_eval_entry
node_eval_entry -->|"mel spectrograms"| node_waveglow_bridge
node_waveglow_bridge -->|"synthesize"| node_waveglow_pkg
node_waveglow_pkg -->|"wav output"| node_sample_outputs

click node_ljspeech_data "https://github.com/xcmyz/fastspeech/tree/master/data"
click node_alignments_zip "https://github.com/xcmyz/fastspeech/blob/master/alignments.zip"
click node_sample_outputs "https://github.com/xcmyz/fastspeech/tree/master/sample"
click node_text_frontend "https://github.com/xcmyz/fastspeech/tree/master/text"
click node_audio_frontend "https://github.com/xcmyz/fastspeech/tree/master/audio"
click node_preprocess_stage "https://github.com/xcmyz/fastspeech/blob/master/preprocess.py"
click node_dataset_loader "https://github.com/xcmyz/fastspeech/blob/master/dataset.py"
click node_ljspeech_reader "https://github.com/xcmyz/fastspeech/blob/master/data/ljspeech.py"
click node_fastspeech_model "https://github.com/xcmyz/fastspeech/blob/master/model.py"
click node_model_modules "https://github.com/xcmyz/fastspeech/blob/master/modules.py"
click node_transformer_pkg "https://github.com/xcmyz/fastspeech/tree/master/transformer"
click node_losses "https://github.com/xcmyz/fastspeech/blob/master/loss.py"
click node_optimizer_cfg "https://github.com/xcmyz/fastspeech/blob/master/optimizer.py"
click node_train_entry "https://github.com/xcmyz/fastspeech/blob/master/train.py"
click node_eval_entry "https://github.com/xcmyz/fastspeech/blob/master/eval.py"
click node_waveglow_pkg "https://github.com/xcmyz/fastspeech/tree/master/waveglow"
click node_waveglow_bridge "https://github.com/xcmyz/fastspeech/blob/master/glow.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ljspeech_data,node_alignments_zip,node_sample_outputs toneBlue
class node_text_frontend,node_audio_frontend toneAmber
class node_preprocess_stage,node_dataset_loader,node_ljspeech_reader toneMint
class node_fastspeech_model,node_model_modules,node_transformer_pkg toneRose
class node_losses,node_optimizer_cfg,node_train_entry,node_eval_entry toneIndigo
class node_waveglow_pkg,node_waveglow_bridge toneTeal
```

## Components

### Data Assets

Raw corpus, alignments, and example outputs.

- [LJSpeech data](https://github.com/xcmyz/fastspeech/tree/master/data): raw corpus · Raw audio and transcripts used as the training source. · `data`
- [Alignments](https://github.com/xcmyz/fastspeech/blob/master/alignments.zip): duration targets · Phoneme-to-frame durations used for supervised FastSpeech training. · `alignments.zip`
- [Samples](https://github.com/xcmyz/fastspeech/tree/master/sample): generated audio · Example synthesized waveforms produced by the inference pipeline. · `sample`

### Front-Ends

Text and audio conversion layers.

- [Text](https://github.com/xcmyz/fastspeech/tree/master/text): normalization · Text normalization, number expansion, symbol conversion, and dictionary lookup. · `text`
- [Audio](https://github.com/xcmyz/fastspeech/tree/master/audio): feature utils · STFT, mel-spectrogram, and waveform processing utilities. · `audio`

### Preprocessing

Offline feature generation and dataset assembly.

- [Preprocess](https://github.com/xcmyz/fastspeech/blob/master/preprocess.py): offline build · Builds supervised training artifacts from audio, text, and alignments. · `preprocess.py`
- [Dataset](https://github.com/xcmyz/fastspeech/blob/master/dataset.py): batch loader · Loads precomputed text, mel, and duration tensors for training. · `dataset.py`
- [LJSpeech IO](https://github.com/xcmyz/fastspeech/blob/master/data/ljspeech.py): dataset bridge · Dataset-specific helpers for reading LJSpeech-derived artifacts. · `data/ljspeech.py`

### Acoustic Model

FastSpeech network and supporting transformer internals.

- [FastSpeech](https://github.com/xcmyz/fastspeech/blob/master/model.py): acoustic model · Core duration-based text-to-mel synthesis model. · `model.py`
- [Modules](https://github.com/xcmyz/fastspeech/blob/master/modules.py): model blocks · Reusable building blocks used by the FastSpeech architecture. · `modules.py`
- [Transformer](https://github.com/xcmyz/fastspeech/tree/master/transformer): layers package · Vendored attention, feed-forward, and layer primitives. · `transformer`

### Training

Losses, optimization, and runtime entry points.

- [Loss](https://github.com/xcmyz/fastspeech/blob/master/loss.py): training objective · Supervises mel prediction and duration prediction. · `loss.py`
- [Optimizer](https://github.com/xcmyz/fastspeech/blob/master/optimizer.py): update rule · Optimization setup used during training. · `optimizer.py`
- [Train](https://github.com/xcmyz/fastspeech/blob/master/train.py): entry point · Main training loop with checkpointing and logging. · `train.py`
- [Eval](https://github.com/xcmyz/fastspeech/blob/master/eval.py): entry point · Loads checkpoints and produces mel predictions for synthesis. · `eval.py`

### Vocoder

Mel-to-waveform synthesis boundary.

- [WaveGlow](https://github.com/xcmyz/fastspeech/tree/master/waveglow): vocoder package · Mel-to-waveform synthesis package used as the audio rendering boundary. · `waveglow`
- [Vocoder bridge](https://github.com/xcmyz/fastspeech/blob/master/glow.py): inference helper · Glue code that converts predicted mels into audio with WaveGlow. · `glow.py`

## Connections

- LJSpeech data → Preprocess: raw audio/text · Source corpus is consumed during offline preprocessing.
- Alignments → Preprocess: durations · Alignment targets are converted into supervised duration data.
- Text → Preprocess: clean text · Normalized symbol sequences are built from transcripts.
- Audio → Preprocess: mel features · Audio utilities produce mel/STFT features for training artifacts.
- Preprocess → Dataset: stored features · Offline outputs are loaded back as training examples.
- Dataset → LJSpeech IO: LJSpeech batches · Dataset-specific access shapes batch inputs and targets.
- Dataset → Train: mini-batches · Batched tensors feed the training loop.
- LJSpeech IO → Train: dataset access · Dataset helpers support the training pipeline.
- Text → FastSpeech: token ids · Text normalization feeds tokenized inputs to the model.
- FastSpeech → Modules: uses · Core architecture is assembled from reusable blocks.
- FastSpeech → Transformer: layers · Transformer primitives provide attention and feed-forward internals.
- FastSpeech → Loss: predictions · Model outputs are compared against mel and duration targets.
- Loss → Optimizer: gradients · Training objective drives optimization updates.
- Optimizer → Train: step updates · Optimizer config is applied inside the training loop.
- Train → FastSpeech: fit · Training updates the acoustic model parameters.
- Train → Eval: checkpoint · Trained checkpoints are consumed by evaluation.
- Eval → Vocoder bridge: mel spectrograms · Evaluation emits predicted mels for waveform rendering.
- Vocoder bridge → WaveGlow: synthesize · Bridge calls the WaveGlow vocoder package.
- WaveGlow → Samples: wav output · Generated example audio is written as sample files.