# bshall/knn-vc architecture

> Architecture diagram of the GitHub repository bshall/knn-vc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bshall/knn-vc
- Repository: https://github.com/bshall/knn-vc

## Overview

##### Project type and purpose
- This is an open-source voice-conversion toolkit (kNN-VC) implemented in Python. It provides both inference (via Torch Hub) and training code for its components. The pipeline follows an encoder–converter–vocoder pattern:
- Encoder: pretrained WavLM (self-supervised speech features)
- Converter: non-parametric k-Nearest Neighbors matching
- Vocoder: HiFi-GAN adapted to generate waveforms from WavLM features

##### File structure
- Top-level scripts and integrations:
- hubconf.py – Torch Hub entry point for inference
- matcher.py – Core kNN matching logic and model wrapper
- prematch_dataset.py – Utility to precompute and optionally “prematch” features for training
- knnvc_demo.ipynb & knnvc_utils.py – Jupyter demo / helper code
- Data directories:
- data_splits/ – CSV files defining training/validation splits for HiFi-GAN training
- Component directories:
- wavlm/ – Wrappers and modules for the WavLM encoder (imported/copied from Microsoft’s repo)
- hifigan/ – Adapted HiFi-GAN code for vocoding WavLM features, including:
– config_v1_wavlm.json – model settings
– meldataset.py – dataset reader & feature loader
– models.py – generator & discriminator definitions
– train.py – training entry point
– utils.py – auxiliary functions for both inference and training

##### README for additional insights
- The README describes the kNN-VC pipeline (see figure “kNN-VC setup”) and outlines:
- Inference steps (load via torch.hub, get features, match, vocode)
- Training steps (precompute features with prematch_dataset.py, then train HiFi-GAN with train.py)
- Dependencies: torch, torchaudio, numpy for inference; librosa, tensorboard, etc., for training.
- Architectural note: WavLM is frozen, kNN is non-learning, only HiFi-GAN requires training.

1. Audio I/O
2. Feature Extractor (Encoder)
– WavLM (wavlm/WavLM.py + modules.py)
3. Converter
– kNN matching logic (matcher.py)
– prematch_dataset.py (batch precomputation for training)
4. Vocoder
– HiFi-GAN (hifigan/models.py + train.py + utils.py + meldataset.py + config)
5. Data Storage
– data_splits CSVs, pretrained checkpoints (Torch Hub / Releases)
6. Integration / API
– hubconf.py for inference
– knnvc_demo.ipynb as usage example

- For inference:
[Source WAV] → (WavLM Encoder) → Feature Vector → (matcher.py) with Reference Features → kNN output features → (HiFi-GAN Vocoder) → [Output WAV]
Reference WAVs → (WavLM Encoder) → Reference Features → stored in memory or prematched lookup
- For training HiFi-GAN:
[Librispeech WAVs] → (prematch_dataset.py + WavLM) → Precomputed feature files → (hifigan/train.py + meldataset.py) → Model checkpoints → (data_splits define train/val)

c. Architectural patterns or principles
- Modular pipeline: each stage (encode, convert, vocode) is independently swappable
- Non-parametric converter (kNN): no backprop required
- Frozen encoder: reuse a strong pretrained model
- Adapted external components: WavLM and HiFi-GAN are imported and lightly modified
- Data-driven training: separate data preprocessing and model training steps

d. Technologies, frameworks, and libraries
- PyTorch & Torch Hub
- torchaudio, numpy (inference)
- librosa, tensorboard, matplotlib, scipy (training)
- JSON config for HiFi-GAN hyperparameters
- Jupyter notebook for demos

##### Elements
- Blue rectangles for processing modules (Encoder, Converter, Vocoder)
- Orange parallelograms for external integrations (Torch Hub, Jupyter Notebook)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Inference Pipeline
    subgraph "Audio I/O"
        direction TB
        SourceWav["Source WAV\n(File)"]:::file
        RefWav["Reference WAV\n(File)"]:::file
        OutWav["Output WAV\n(File)"]:::file
    end

    subgraph "Processing Modules"
        direction TB
        Encoder["WavLM Encoder"]:::process
        Converter["kNN Matcher"]:::process
        Vocoder["HiFi-GAN Vocoder"]:::process
    end

    subgraph "Integration / API"
        direction TB
        TorchHub["Torch Hub\n(hubconf.py)"]:::external
        Demo["Jupyter Demo\n(knnvc_demo.ipynb)"]:::external
    end

    subgraph "Data Storage"
        direction TB
        DataSplits["Train/Valid Splits\n(CSV)"]:::storage
        Checkpoints["Pretrained Checkpoints"]:::storage
    end

    %% Inference Flow
    SourceWav -->|"waveform"| Encoder
    RefWav -->|"waveform"| Encoder
    Encoder -->|"features"| Converter
    Converter -->|"matched features"| Vocoder
    Vocoder -->|"waveform"| OutWav
    TorchHub -->|"loads Encoder & Vocoder"| Encoder
    TorchHub -->|"invokes match & vocode"| Vocoder
    Demo -->|"example usage"| TorchHub

    %% Training Pipeline
    subgraph "Training Flow"
        direction TB
        LibriWav["Librispeech WAVs\n(File)"]:::file
        Prematch["Precompute Dataset\n(prematch_dataset.py)"]:::process
        PreFeat["Precomputed Features\n(File)"]:::storage
        TrainScript["HiFi-GAN Training\n(hifigan/train.py)"]:::process
        ModelCKPT["Model Checkpoints"]:::storage

        LibriWav -->|"waveform"| Prematch
        Prematch -->|"features"| PreFeat
        PreFeat -->|"features"| TrainScript
        DataSplits -.-> TrainScript
        TrainScript -->|"checkpoints"| ModelCKPT
        ModelCKPT -->|"export"| Checkpoints
        Checkpoints --> TorchHub
    end

    %% Click Events
    click Encoder "https://github.com/bshall/knn-vc/blob/master/wavlm/WavLM.py"
    click Converter "https://github.com/bshall/knn-vc/blob/master/matcher.py"
    click Prematch "https://github.com/bshall/knn-vc/blob/master/prematch_dataset.py"
    click TrainScript "https://github.com/bshall/knn-vc/blob/master/hifigan/train.py"
    click Vocoder "https://github.com/bshall/knn-vc/blob/master/hifigan/models.py"
    click DataSplits "https://github.com/bshall/knn-vc/blob/master/data_splits/wavlm-hifigan-train.csv"
    click DataSplits "https://github.com/bshall/knn-vc/blob/master/data_splits/wavlm-hifigan-valid.csv"
    click TorchHub "https://github.com/bshall/knn-vc/blob/master/hubconf.py"
    click Demo "https://github.com/bshall/knn-vc/blob/master/knnvc_demo.ipynb"

    %% Styles
    classDef process fill:#ADD8E6,stroke:#000;
    classDef storage fill:#90EE90,stroke:#000;
    classDef external fill:#FFA500,stroke:#000;
    classDef file fill:#D3D3D3,stroke:#000;
```
