How brian-mwangi-developer/styletts2-prep works
Project type and purpose
• This is a research-grade, full-stack machine-learning pipeline for text-to-speech (TTS) synthesis (“StyleTTS2”).
• It implements a two-stage training pipeline: (1) latent-style diffusion TTS network and (2) adversarial fine-tuning with large speech-language model (SLM) discriminators.
• It supports data preprocessing (text alignment, pitch extraction, PL-BERT text embedding), model training, inference notebooks, a simple HTTP API and Docker deployment.
High-level components and directories
• Data layer (folder “Data”): raw .wav files, train/val lists, OOD texts.
• Config management (“Configs”): YAML configs for each dataset and finetuning.
• Preprocessing utilities (“Utils/ASR”, “Utils/JDC”, “Utils/PLBERT”, “text_utils.py”): text aligner, pitch extractor, BERT embeddings, phonemizer.
• Core model modules (“Modules”):
– diffusion/…: latent diffusion sampler, denoiser, utilities
– discriminators.py (SLM adversarial losses)
– hifigan.py (vocoder), istftnet.py (inverse STFT vocoder)
– slmadv.py (adversarial wrapper)
– utils.py (common functions)
• Training orchestration (root):
– train_first.py (stage-1 training via Accelerate)
– train_second.py (stage-2 adversarial fine-tuning, DP fallback)
– train_finetune(.py, _accelerate.py): finetuning on new speakers
• Inference and API:
– libri_inference.py, api.py (lightweight HTTP inference server)
– Demo notebooks (Colab + Jupyter in “Demo”)
– Dockerfile.api (serving), Dockerfile.train (training environment)
• Models (“Models”): checkpoints, sample outputs, TensorBoard logs
Relationships & data flow
1. Raw data → Preprocessor:
• Text aligner (ASR) → phoneme durations
• Pitch extractor (JDC) → F0 contour
• PL-BERT (text embedding) → contextual text features
2. Stage-1 TTS network:
• Text embeddings + durations + F0 → diffusion-based style sampler → mel-spectrogram decoder
• Losses: reconstruction (L1/L2), alignment, diffusion denoising
3. Stage-2 adversarial training:
• Generated mel → vocoder (HiFiGAN or iSTFTNet) → waveform
• Discriminators: large pre-trained SLMs (WavLM) + HifiGAN critics → adversarial loss backprop
4. Inference pipeline:
Text input → phonemizer → PL-BERT → style diffusion → mel decoder → vocoder → audio → API endpoint/notebook
Architectural patterns & principles
• Modular layers separating preprocessing, model definition, training scripts, inference/API.
• Two-stage pipeline: diffusion generator + adversarial fine-tuning (sequential stages).
• Pluggable pretrained modules (ASR, JDC, PL-BERT) loaded as external dependencies.
• Config-driven orchestration for datasets, hyperparameters, multi- vs. single-speaker mode.