How jaywalnut310/vits works
Project Type and Purpose
• VITS is an end-to-end text-to-speech (TTS) system: a single-stage, parallelizable neural TTS model combining variational inference (VAE), normalizing flows, and adversarial training.
• Its primary purpose is to convert raw text into natural-sounding speech with diverse prosody.
File Structure
• Top-level Python scripts:
– train.py & train_ms.py: launch the training pipelines for single-speaker (LJ) and multi-speaker (VCTK).
– inference.ipynb: demonstrates the inference pipeline.
• Core modules (modules.py, models.py, attentions.py, commons.py, losses.py, transforms.py, mel_processing.py, data_utils.py, utils.py): implement encoders, decoders, flows, discriminators, feature transforms, loss functions, data loaders.
• preprocess.py & monotonic_align/: the G2P-and-duration preprocessing package (Cython-accelerated alignment).
• configs/*.json: hyperparameter definitions for different datasets.
• filelists/: train/val/test splits for LJ and VCTK.
• text/: text cleaning & symbol definitions.
• resources/: embedded architecture figures (training vs. inference).
README for Insights
• The text emphasizes:
– Encoder (text → posterior encoder + normalizing flows)
– Stochastic duration predictor
– Generator (decoder)
– Discriminators (multi-scale, multi-period)
– VAE latent sampling (KL loss)
– GAN-style adversarial loss
– Separate training vs. inference data flow.
Main Components & Their Relationships
A. Data Ingestion & Preprocessing
• Raw text & waveform datasets (LJ Speech, VCTK)
• Text cleaning (text/cleaners.py), G2P, phoneme indexing
• Monotonic alignment search (monotonic_align): align text with mel frames
• Mel-spectrogram extraction (mel_processing.py)
• Filelists drive batching.
B. Model Architecture (Training Mode)
1. Text Encoder → produces hidden text features
2. Posterior Encoder (mel→latent) + Normalizing Flows → posterior latent z
3. Stochastic Duration Predictor → predicts durations for text frames
4. Generator (Decoder) → converts z to waveform
5. Discriminators → adversarial feedback
6. Losses: reconstruction (L1/L2), KL divergence, adversarial, duration, feature matching
C. Model Architecture (Inference Mode)
1. Text Encoder → text features
2. Stochastic Duration Predictor → durations
3. Prior sampling + Normalizing Flows inversely → latent z
4. Generator → waveform output
D. Utilities & Scripts
• train.py / train_ms.py orchestrate data preload, model instantiation, optimization loops
• Inference notebook runs the above inference pipeline
• configs/*.json supply hyperparameters to scripts.
Architectural Patterns & Technologies
• Pattern: VAE + Normalizing Flows + GAN (adversarial training) in a single unified network.
• Modularization: clear separation of text frontend, posterior/prior networks, duration model, generator, discriminators.
• Data pipeline: Python + PyTorch DataLoaders, Cython for alignment speedup.
• CLI/config-driven experimentation (JSON configs, argparse).
• Dependencies: Python ≥3.6, PyTorch, librosa, numpy, Cython.
Final Checklist
• Two panels: Training vs. Inference
• Clear labels for:
– Data sources (LJ / VCTK wav folders)
– Preprocessing modules
– Neural network components (encoder, flow, decoder, discriminator)
– Scripts/orchestration