How ntt123/light-speed works
Purpose and entry points
Light Speed is a VITS-derived text-to-speech training and inference repository. Its distinguishing training path uses externally aligned phoneme durations to expand phoneme representations to acoustic frame resolution before synthesis.
Primary entry points:
• Single- or multi-GPU training: train.py
• Speech generation workflow: inference.ipynb
• Dataset preparation: prepare_ljs_tfdata.ipynb, prepare_vbx_tfdata.ipynb
• Duration-model training: train_duration_model.ipynb
Data preparation and alignment boundary
Training data preparation is notebook-driven and produces TensorFlow-oriented dataset artifacts consumed by the loader layer:
• prepare_ljs_tfdata.ipynb
• prepare_vbx_tfdata.ipynb
• tfloader.py
An external forced aligner, such as Montreal Forced Aligner, is required upstream to obtain phoneme-duration ground truth. This is a material boundary: alignment quality and phoneme/token conventions must remain compatible with the prepared dataset and model inputs.
Training orchestration
train.py is the training control plane. It coordinates data loading, model construction, losses, optimization/checkpoint behavior, and distributed execution. The documented runtime supports standard Python execution and PyTorch distributed launch via torchrun.
Key supporting modules:
• train.py
• tfloader.py
• losses.py
• config.json
config.json is the likely shared source for model/training dimensions and runtime parameters; changes must remain consistent with dataset feature shapes and model submodules.
Acoustic model and duration-conditioned expansion
The central neural architecture is organized in:
• models.py
• modules.py
• attentions.py
The model consumes phoneme-level information and aligned durations, then upsamples the phoneme sequence to frame-level conditioning. This replaces or reduces reliance on learned duration alignment during the primary TTS training path, favoring robustness over some potential speech-quality tradeoff.
Attention and reusable neural building blocks are separated from top-level model assembly, making models.py the integration boundary and modules.py / attentions.py the primary reusable architecture layers.
Latent flow and waveform/acoustic transformations
The VITS-style latent transformation path is represented by:
• flow.py
• modules.py
• models.py
The repository specifically modifies the WaveNet flow receptive field with dilated convolutions to capture longer-range interactions. Flow construction and its coupling with encoders/decoders should therefore be treated as a coordinated architecture: tensor channel counts, sequence masks, and latent dimensions must agree across these files.
Mel-spectrogram and related audio feature processing are isolated in:
• mel_processing.py
• commons.py
This forms the boundary between raw/audio-derived training targets and the neural model’s acoustic inputs or losses.
Inference and deployment dependencies
Inference is demonstrated rather than exposed as a standalone service:
• inference.ipynb
• models.py
• mel_processing.py
A deployed demo/checkpoint consumer must reproduce preprocessing, phoneme representation, duration prediction or supplied durations, and checkpoint configuration used during training. Pretrained voice demos are hosted externally on Hugging Face Spaces; the repository itself contains training and notebook-based inference workflows rather than service infrastructure.
Main entry points: models.py VITS model assembly · train.py Training control plane · inference.ipynb Speech generation · config.json Shared configuration
Components 18
Data & alignment
- External forced aligner · Produces phoneme-duration ground truth compatible with dataset tokens.
- LJS dataset preparation
prepare_ljs_tfdata.ipynb - VBX dataset preparation
prepare_vbx_tfdata.ipynb - Prepared TF datasets · Phoneme, duration, and acoustic training features.
- Dataset loader
tfloader.py
Training
- Shared configuration · Dimensions and runtime settings must match data and model shapes.
config.json - Training control plane · Coordinates loading, model construction, losses, optimization, checkpoints, and distributed execution.
train.py - PyTorch distributed launch · Optional multi-GPU execution via torchrun.
- Training losses
losses.py
Duration-conditioned VITS
- Mel & audio features · Converts audio-derived targets and acoustic features for model training.
mel_processing.py - Shared tensor utilities
commons.py - VITS model assembly · Connects encoders, duration-conditioned expansion, flow, and decoder.
models.py - Aligned duration expansion · Upsamples phoneme representations to acoustic frame resolution using aligned durations.
- Latent flow · WaveNet-style dilated-convolution flow captures longer-range interactions.
flow.py - Reusable neural modules
modules.py - Attention layers
attentions.py
Inference
- Duration-model training · Trains duration prediction for inference when durations are not supplied.
train_duration_model.ipynb - Speech generation · Reproduces training preprocessing, phonemes, durations, and checkpoint configuration.
inference.ipynb
Connections 22
- External forced aligner → LJS dataset preparation: aligned durations
- External forced aligner → VBX dataset preparation: aligned durations
- LJS dataset preparation → Prepared TF datasets: writes
- VBX dataset preparation → Prepared TF datasets: writes
- Prepared TF datasets → Dataset loader: consumed by
- Dataset loader → Training control plane: batches
- Shared configuration → Training control plane: runtime settings
- PyTorch distributed launch → Training control plane: launches
- Training control plane → VITS model assembly: builds and optimizes
- Mel & audio features → Training control plane: acoustic targets
- Training losses → Training control plane: objective
- Shared configuration → VITS model assembly: model dimensions
- VITS model assembly → Aligned duration expansion: uses
- Aligned duration expansion → Latent flow: frame-level conditioning
- Latent flow → Reusable neural modules: built from
- VITS model assembly → Reusable neural modules: composes
- VITS model assembly → Attention layers: composes
- Shared tensor utilities → Mel & audio features: shared operations
- Duration-model training → Speech generation: duration predictor
- VITS model assembly → Speech generation: checkpointed model
- Mel & audio features → Speech generation: preprocessing
- Shared configuration → Speech generation: matching configuration