How bshall/knn-vc works
Project type and purpose
• This is an open-source voice-conversion toolkit (kNN-VC) implemented in Python. It provides both inference (via Torch Hub) and training code for its components. The pipeline follows an encoder–converter–vocoder pattern:
• Encoder: pretrained WavLM (self-supervised speech features)
• Converter: non-parametric k-Nearest Neighbors matching
• Vocoder: HiFi-GAN adapted to generate waveforms from WavLM features
File structure
• Top-level scripts and integrations:
• hubconf.py – Torch Hub entry point for inference
• matcher.py – Core kNN matching logic and model wrapper
• prematch_dataset.py – Utility to precompute and optionally “prematch” features for training
• knnvc_demo.ipynb & knnvc_utils.py – Jupyter demo / helper code
• Data directories:
• data_splits/ – CSV files defining training/validation splits for HiFi-GAN training
• Component directories:
• wavlm/ – Wrappers and modules for the WavLM encoder (imported/copied from Microsoft’s repo)
• hifigan/ – Adapted HiFi-GAN code for vocoding WavLM features, including:
– config_v1_wavlm.json – model settings
– meldataset.py – dataset reader & feature loader
– models.py – generator & discriminator definitions
– train.py – training entry point
– utils.py – auxiliary functions for both inference and training
README for additional insights
• The README describes the kNN-VC pipeline (see figure “kNN-VC setup”) and outlines:
• Inference steps (load via torch.hub, get features, match, vocode)
• Training steps (precompute features with prematch_dataset.py, then train HiFi-GAN with train.py)
• Dependencies: torch, torchaudio, numpy for inference; librosa, tensorboard, etc., for training.
• Architectural note: WavLM is frozen, kNN is non-learning, only HiFi-GAN requires training.
1. Audio I/O
2. Feature Extractor (Encoder)
– WavLM (wavlm/WavLM.py + modules.py)
3. Converter
– kNN matching logic (matcher.py)
– prematch_dataset.py (batch precomputation for training)
4. Vocoder
– HiFi-GAN (hifigan/models.py + train.py + utils.py + meldataset.py + config)
5. Data Storage
– data_splits CSVs, pretrained checkpoints (Torch Hub / Releases)
6. Integration / API
– hubconf.py for inference
– knnvc_demo.ipynb as usage example
• For inference:
[Source WAV] → (WavLM Encoder) → Feature Vector → (matcher.py) with Reference Features → kNN output features → (HiFi-GAN Vocoder) → [Output WAV]
Reference WAVs → (WavLM Encoder) → Reference Features → stored in memory or prematched lookup
• For training HiFi-GAN:
[Librispeech WAVs] → (prematch_dataset.py + WavLM) → Precomputed feature files → (hifigan/train.py + meldataset.py) → Model checkpoints → (data_splits define train/val)
c. Architectural patterns or principles
• Modular pipeline: each stage (encode, convert, vocode) is independently swappable
• Non-parametric converter (kNN): no backprop required
• Frozen encoder: reuse a strong pretrained model
• Adapted external components: WavLM and HiFi-GAN are imported and lightly modified
• Data-driven training: separate data preprocessing and model training steps
d. Technologies, frameworks, and libraries
• PyTorch & Torch Hub
• torchaudio, numpy (inference)
• librosa, tensorboard, matplotlib, scipy (training)
• JSON config for HiFi-GAN hyperparameters
• Jupyter notebook for demos
Elements
• Blue rectangles for processing modules (Encoder, Converter, Vocoder)
• Orange parallelograms for external integrations (Torch Hub, Jupyter Notebook)