# nam2speech/nam2speech-code architecture

> Architecture diagram of the GitHub repository nam2speech/nam2speech-code, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nam2speech/nam2speech-code
- Repository: https://github.com/nam2speech/nam2speech-code

## Overview

##### Project type and purpose
- This is a research-oriented, machine-learning pipeline for NAM-to-Speech synthesis. It orchestrates data download, self-supervised feature extraction (HuBERT), two model training stages (HiFi-GAN vocoder and Seq2Seq converter), inference, and evaluation (WER/CER).

##### High-level architecture overview
Break the system into five main layers/modules:
a. Data Acquisition & Storage
b. Feature Extraction (HuBERT units & features)
c. Model Training
- Vocoder (HiFi-GAN)
- Seq2Seq converter
d. Inference Pipelines
- Vocoder inference
- Seq2Seq inference + downstream vocoder
e. Evaluation

##### Key components and their relationships
- Raw Data Sources
- CSTR NAM TIMIT Plus corpus (`runs/data/NAM`, `runs/data/whisper`)
- LJSpeech corpus (`runs/data/LJSpeech`)
- External services (Google Drive links, fairseq checkpoints)
- Download Script (`download.py`)
- Downloads raw corpora into `runs/data/`
- HuBERT Extraction Module (`utils/hubert_extraction/`)
- Config files: `hubert_config.yaml`, `unit_config.yaml`
- Scripts: `extractor.py` (produces `.txt` unit sequences), `unit_extractor.py` (derives units from predicted embeddings)
- Vocoder Module (`utils/vocoder/`)
- Configs: `LJSpeech_config.json`, `NAM_config.json`
- Preprocessor: `preprocessor.py` → prepares training/val splits
- Trainer: `train.py` → HiFi-GAN distributed training
- Inference: `inference.py` → waveform synthesis from codebooks
- Models/utilities under `models.py`, `dataset.py`, `utils.py`
- Seq2Seq Module (`utils/Seq2Seq/`)
- Config: `Seq2Seq_config.yaml`
- Preprocessor: `preprocessor.py` → prepares inputs (HuBERT units → embeddings)
- Train script: top-level `train.py`
- Inference script: top-level `inference.py`
- Core ML Helpers (`modules/`)
- Data handling (`data.py`), loss functions (`loss.py`), FFT routines (`fft.py`), parrot module (`parrot.py`)

Data flow summary:
Raw audio → HuBERT extraction → unit/feature files →
a) Vocoder preprocessing → Vocoder training → Vocoder checkpoints → Vocoder inference
b) Seq2Seq preprocessing → Seq2Seq training → Seq2Seq inference → predicted embeddings → unit_extractor → predicted units → Vocoder inference
Final waveforms → WER/CER evaluation (`wer.py`).

##### Architectural patterns & design principles
- Pipeline pattern: strict linear data flow through multiple stages
- Modular separation: each stage lives in its own sub-package with clear config boundaries
- Config-driven reproducibility: all parameters in YAML/JSON files
- Distributed training support via `torch.distributed.run`
- Reusability: `modules/` contains shared utilities

##### Technologies & frameworks
- Python 3.8, PyTorch
- fairseq for HuBERT feature extraction
- HiFi-GAN vocoder
- Self-supervised HuBERT (pretrained checkpoints)
- YAML/JSON for configuration
- Conda environment management

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Sources
    subgraph "External Sources"
        direction TB
        GDrive["Google Drive Links"]:::external
        Fairseq["fairseq Checkpoints"]:::external
    end

    %% Data Acquisition
    Download["download.py – Download Script"]:::compute
    RawData["Raw Audio\n(runs/data/)"]:::data

    GDrive -->|download| Download
    Fairseq -->|download| Download
    Download -->|.wav| RawData

    %% HuBERT Extraction
    subgraph "HuBERT Extraction Module"
        direction TB
        HE_Config1["hubert_config.yaml"]:::config
        HE_Config2["unit_config.yaml"]:::config
        Extractor["extractor.py\n(produces .txt units)"]:::compute
        UnitExtractor["unit_extractor.py"]:::compute
    end

    RawData -->|.wav| Extractor
    HE_Config1 --> Extractor
    Extractor -->|.txt units| UnitFiles["Unit Files"]:::data
    HE_Config2 --> UnitExtractor
    UnitExtractor -->|.txt units| UnitFiles

    %% Branching to pipelines
    UnitFiles --> VocoderPre
    UnitFiles --> Seq2SeqPre

    %% Vocoder Pipeline
    subgraph "Vocoder Module (HiFi-GAN)"
        direction TB
        Vocoder_Config1["LJSpeech_config.json"]:::config
        Vocoder_Config2["NAM_config.json"]:::config
        VocoderPre["preprocessor.py"]:::compute
        VocoderTrain["train.py\n(HiFi-GAN Trainer)"]:::compute
        VocoderInfer["inference.py\n(Waveform Synthesis)"]:::compute
        Models["models.py"]:::compute
        Dataset["dataset.py"]:::compute
        Utils["utils.py"]:::compute
    end

    Vocoder_Config1 --> VocoderPre
    Vocoder_Config2 --> VocoderPre
    UnitFiles -->|.txt units| VocoderPre
    VocoderPre -->|processed data| VocoderTrain
    VocoderTrain -->|.ckpt| Checkpoints["Vocoder Checkpoints"]:::data
    Checkpoints -->|.ckpt| VocoderInfer
    VocoderInfer -->|.wav| VocoderOutput["Generated Speech"]:::data

    %% Seq2Seq Pipeline
    subgraph "Seq2Seq Module"
        direction TB
        Seq2Seq_Config["Seq2Seq_config.yaml"]:::config
        Seq2SeqPre["preprocessor.py"]:::compute
        Seq2SeqTrain["train.py\n(Seq2Seq Trainer)"]:::compute
        Seq2SeqInfer["inference.py\n(Seq2Seq Inference)"]:::compute
    end

    Seq2Seq_Config --> Seq2SeqPre
    UnitFiles -->|.txt units| Seq2SeqPre
    Seq2SeqPre --> Seq2SeqTrain
    Seq2SeqTrain --> Seq2SeqInfer
    Seq2SeqInfer -->|embeddings| Embeddings["Predicted Embeddings"]:::data
    Embeddings --> UnitExtractor
    UnitExtractor -->|.txt units| Seq2SeqUnits["Predicted Units"]:::data
    Seq2SeqUnits --> |.txt units| VocoderInfer

    VocoderInfer -->|.wav| Seq2SeqOutput["Seq2Seq Generated Speech"]:::data

    %% Shared Utilities
    subgraph "Core ML Helpers (modules/)"
        direction LR
        DataUtils["data.py"]:::compute
        Loss["loss.py"]:::compute
        FFT["fft.py"]:::compute
        Parrot["parrot.py"]:::compute
    end

    DataUtils --- Extractor
    Loss --- Seq2SeqTrain
    FFT --- VocoderTrain
    Parrot --- Seq2SeqInfer

    %% Evaluation
    Wer["wer.py – Evaluation (WER/CER)"]:::compute
    VocoderOutput --> Wer
    Seq2SeqOutput --> Wer
    Wer --> Reports["Evaluation Reports"]:::data

    %% Click Events
    click Download "https://github.com/nam2speech/nam2speech-code/blob/main/download.py"
    click Extractor "https://github.com/nam2speech/nam2speech-code/blob/main/utils/hubert_extraction/extractor.py"
    click UnitExtractor "https://github.com/nam2speech/nam2speech-code/blob/main/utils/hubert_extraction/unit_extractor.py"
    click HE_Config1 "https://github.com/nam2speech/nam2speech-code/blob/main/utils/hubert_extraction/hubert_config.yaml"
    click HE_Config2 "https://github.com/nam2speech/nam2speech-code/blob/main/utils/hubert_extraction/unit_config.yaml"
    click VocoderPre "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/preprocessor.py"
    click VocoderTrain "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/train.py"
    click VocoderInfer "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/inference.py"
    click Models "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/models.py"
    click Dataset "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/dataset.py"
    click Utils "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/utils.py"
    click Vocoder_Config1 "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/LJSpeech_config.json"
    click Vocoder_Config2 "https://github.com/nam2speech/nam2speech-code/blob/main/utils/vocoder/NAM_config.json"
    click Seq2SeqPre "https://github.com/nam2speech/nam2speech-code/blob/main/utils/Seq2Seq/preprocessor.py"
    click Seq2Seq_Config "https://github.com/nam2speech/nam2speech-code/blob/main/utils/Seq2Seq/Seq2Seq_config.yaml"
    click Seq2SeqTrain "https://github.com/nam2speech/nam2speech-code/blob/main/train.py"
    click Seq2SeqInfer "https://github.com/nam2speech/nam2speech-code/blob/main/inference.py"
    click DataUtils "https://github.com/nam2speech/nam2speech-code/blob/main/modules/data.py"
    click Loss "https://github.com/nam2speech/nam2speech-code/blob/main/modules/loss.py"
    click FFT "https://github.com/nam2speech/nam2speech-code/blob/main/modules/fft.py"
    click Parrot "https://github.com/nam2speech/nam2speech-code/blob/main/modules/parrot.py"
    click Wer "https://github.com/nam2speech/nam2speech-code/blob/main/wer.py"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#4A90E2,stroke-width:1px,shape:cylinder
    classDef compute fill:#E6FFEA,stroke:#28A745,stroke-width:1px,shape:rect
    classDef config fill:#FFF5BA,stroke:#FFC107,stroke-width:1px,shape:rect
    classDef external fill:#F0F0F0,stroke:#888888,stroke-width:1px,shape:cloud
```
