How rickyl-2000/alignsts works
Project Type and Purpose
• AlignSTS is a research-grade speech-to-singing conversion system implemented in PyTorch.
• It provides both training and inference pipelines for an acoustic model (“AlignSTS”) and a neural vocoder (HiFi-GAN or Parallel WaveGAN).
File Structure
• configs/: YAML configuration files for different tasks (singing, speech2singing, TTS).
• data_gen/: Data preprocessing and binarization scripts for singing, speech2singing, and TTS.
• modules/: Core neural network components grouped by model type (diffusion, fastspeech, hifigan, parallel_wavegan, speech2singing).
• vocoders/: Wrapper classes for neural vocoders (HiFi-GAN, PWG).
• tasks/: Orchestrator scripts (tasks/run.py) and task definitions for training/inference of speech2singing and TTS.
• utils/: Common utilities (audio I/O, checkpointing, distributed training, plotting, text processing, etc.).
• usr/: User entry points for evaluation and custom scripts.
README for Insights
• Inference pipeline steps:
1. Prepare checkpoints (checkpoints/alignsts, checkpoints/hifigan).
2. Prepare dataset (binary stats + WAV files).
3. Run tasks/run.py with the proper config.
4. Model outputs mel-spectrograms → Vocoder generates waveforms.
• External dependencies: PyTorch, specific repos (NATSpeech, DiffSinger, ProDiff, SpeechSplit2) used as code references.
Main Components and Their Interactions
A. Configuration Manager
• Loads YAMLs from configs/ → Provides hyperparameters to tasks and modules.
B. Data Preprocessing
• Scripts in data_gen/ → Read raw audio + text → Extract features (e.g., mel, pitch) → Produce binary datasets.
C. Task Orchestrator
• Entry point: tasks/run.py → Reads config → Instantiates Trainer or Inference engine.
• Sub-tasks: tasks/speech2singing/*, tasks/tts/*.
D. Model Modules
• Acoustic model (AlignSTS): code in modules/speech2singing/*.
• Auxiliary models: diffusion (modules/diffusion), fastspeech (modules/fastspeech), transformer TTS.
• Neural vocoder: modules/hifigan, modules/parallel_wavegan & wrapper in vocoders/.
E. Utilities
• Logging, checkpointing (utils/ckpt_utils.py), distributed training (utils/ddp_utils.py), schedulers, plotting, text normalization, etc.
F. External Services (optional)
• ASR or alignment tools referenced in configs (MFA, asr.yaml).
Data Flow (Inference Example):
Raw Speech WAV → Data Preprocessing (feature extraction)
↓
Binary Dataset → Task Orchestrator (infer mode)
↓
AlignSTS Acoustic Model → Predicted Mel-Spectrogram
↓
Neural Vocoder (HiFi-GAN / PWG) → Output Singing WAV
Architectural Patterns & Design Principles
• Modular Design: Separate modules for each model type, enabling easy swapping (e.g., HiFi-GAN vs. PWG).
• Config-Driven Pipeline: YAML files centrally define hyperparameters, model choices, and dataset paths.
• Task Abstraction: base_task.py defines common training/inference behaviors; specialized in sub-packages.
• Utility Layer: A thin wrapper for I/O, logging, distributed training keeps core logic clean.
Final Checklist
• All major directories/components represented.
• Clear data flow from raw input → preprocessing → model → output.
• Configuration management shown centrally.
• External dependencies or services noted.