How rongjiehuang/generspeech works
Project Type and Purpose
• This repository implements a research-grade Text-to-Speech (TTS) system (“GenerSpeech”) in PyTorch, focused on zero-shot style transfer for out-of-domain voices.
• It provides data preparation pipelines, model definitions, training scripts, and inference utilities (including a Gradio demo).
File Structure Analysis
Top-level directories and roles:
• assets: example audio/text for quick inference tests
• data_gen/tts: multi-step data preprocessing (text normalization, wav/text processors), forced-alignment (MFA) training, and binary data binarization
• egs: example dataset configs (LibriTTS, ESD, VCTK, LJSpeech) and scripts for data loading
• modules: core model implementations
– modules/GenerSpeech: acoustic model (generspeech.py, glow, prosody, wavenet)
– modules/fastspeech, hifigan, parallel_wavegan: alternative TTS/post-filter/vocoder modules
• inference: standalone inference scripts
– inference/GenerSpeech.py: CLI inference using acoustic model + vocoder
– inference/gradio: web UI for demo
• tasks: high-level entrypoint (tasks/run.py) for training/inference workflows via config files
• utils: shared utilities (audio I/O, checkpointing, distributed training, plotting, text encoding/norm, schedulers)
• vocoders: wrapper interfaces for Hifi-GAN, PWG
Key config & environment files:
• modules/GenerSpeech/config/generspeech.yaml – main acoustic model hyperparameters
• environment.yaml / requirements.txt – Conda/PyPI dependencies
• egs/egs_bases – reusable base YAML configs for various model components
README Insights
• Workflow stages clearly described: preprocess → align (MFA) → binarize → train → infer
• Pretrained checkpoints for acoustic model, vocoder, and emotion encoder
• External dependencies: Wav2Vec2 (ASR for reference feature extraction), Montreal Forced Aligner (MFA)
a. Main Components
1. Data Preparation Layer
• Raw dataset (audio + transcripts)
• Text processors, wav processors → MFA aligner → binarizer → binary dataset
2. Training Layer
• tasks/run.py orchestrates: loads binary dataset → modules/GenerSpeech acoustic model → optional parallel training (DDP) → checkpoint outputs
• Optional vocoder training (Hifi-GAN / PWG) via modules/… and tasks/vocoder
3. Inference Layer
• CLI (inference/GenerSpeech.py) & Gradio UI (inference/gradio/infer.py)
• Loads checkpoints (acoustic + vocoder + emotion encoder) + reference audio → generates mel spectrogram → vocoder → final waveform
4. External Services
• ASR model (Wav2Vec2) for reference feature extraction
• MFA for text-speech alignment
b. Relationships & Data Flows
• Inference: Reference Audio + Config → Acoustic Model → Mel → Vocoder → Output WAV
• Utilities (utils/) feed into both training and inference for logging, checkpointing, distributed setup
c. Architectural Patterns & Principles
• Modularization: clear separation of data pipelines, model definitions, training logic, inference scripts
• Config-driven design: YAML files define hyperparameters and data paths, enabling reproducibility and easy swapping of datasets/configs
• Extensibility points: modules/… folders allow plugging in new architectures (e.g., replace GenerSpeech with new model)
d. Key Technologies
• PyTorch & DistributedDataParallel (utils/ddp_utils.py)
• YAML-based configuration (Hydra-like patterns)
• MFA & Huggingface Wav2Vec2 for alignment and feature extraction
• Gradio for interactive demo