How nam2speech/nam2speech-code works
Project type and purpose
• This is a research-oriented, machine-learning pipeline for NAM-to-Speech synthesis. It orchestrates data download, self-supervised feature extraction (HuBERT), two model training stages (HiFi-GAN vocoder and Seq2Seq converter), inference, and evaluation (WER/CER).
High-level architecture overview
Break the system into five main layers/modules:
a. Data Acquisition & Storage
b. Feature Extraction (HuBERT units & features)
c. Model Training
• Vocoder (HiFi-GAN)
• Seq2Seq converter
d. Inference Pipelines
• Vocoder inference
• Seq2Seq inference + downstream vocoder
e. Evaluation
Key components and their relationships
• Raw Data Sources
• CSTR NAM TIMIT Plus corpus (runs/data/NAM, runs/data/whisper)
• LJSpeech corpus (runs/data/LJSpeech)
• External services (Google Drive links, fairseq checkpoints)
• Download Script (download.py)
• Downloads raw corpora into runs/data/
• HuBERT Extraction Module (utils/hubert_extraction/)
• Config files: hubert_config.yaml, unit_config.yaml
• Scripts: extractor.py (produces .txt unit sequences), unit_extractor.py (derives units from predicted embeddings)
• Vocoder Module (utils/vocoder/)
• Configs: LJSpeech_config.json, NAM_config.json
• Preprocessor: preprocessor.py → prepares training/val splits
• Trainer: train.py → HiFi-GAN distributed training
• Inference: inference.py → waveform synthesis from codebooks
• Models/utilities under models.py, dataset.py, utils.py
• Seq2Seq Module (utils/Seq2Seq/)
• Config: Seq2Seq_config.yaml
• Preprocessor: preprocessor.py → prepares inputs (HuBERT units → embeddings)
• Train script: top-level train.py
• Inference script: top-level inference.py
• Core ML Helpers (modules/)
• Data handling (data.py), loss functions (loss.py), FFT routines (fft.py), parrot module (parrot.py)
Data flow summary:
Raw audio → HuBERT extraction → unit/feature files →
a) Vocoder preprocessing → Vocoder training → Vocoder checkpoints → Vocoder inference
b) Seq2Seq preprocessing → Seq2Seq training → Seq2Seq inference → predicted embeddings → unit_extractor → predicted units → Vocoder inference
Final waveforms → WER/CER evaluation (wer.py).
Architectural patterns & design principles
• Pipeline pattern: strict linear data flow through multiple stages
• Modular separation: each stage lives in its own sub-package with clear config boundaries
• Config-driven reproducibility: all parameters in YAML/JSON files
• Distributed training support via torch.distributed.run
• Reusability: modules/ contains shared utilities
Technologies & frameworks
• Python 3.8, PyTorch
• fairseq for HuBERT feature extraction
• HiFi-GAN vocoder
• Self-supervised HuBERT (pretrained checkpoints)
• YAML/JSON for configuration
• Conda environment management