# ncsoft/phonmatchnet architecture

> Architecture diagram of the GitHub repository ncsoft/phonmatchnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ncsoft/phonmatchnet
- Repository: https://github.com/ncsoft/phonmatchnet

## Overview

##### Project Type and Purpose
- This is a research-oriented ML pipeline (PyTorch) for zero-shot keyword spotting guided by phonemes.
- It ingests audio and text (user-defined keywords), extracts features, fuses them in a neural model, and trains end-to-end.

##### High-Level Components
A. Data Sources
– LibriPhrase dataset
– Google Speech Commands
– Qualcomm Keyword Speech
B. Data-Loading & Preprocessing
– dataset/… modules (g2p for grapheme-to-phoneme, google.py, libriphrase.py, qualcomm.py)
– Feature extraction: model/log_melspectrogram.py
– Speech embeddings: model/google_speech_embedding (TF-Hub SavedModel) & model/speech_embedding.py
C. Model Core
– Extractor (model/extractor.py) – stacks and fuses audio/text features
– Encoder (model/encoder.py) – projection heads for each modality
– Discriminator (model/discriminator.py) – scoring network for keyword vs. background
– UKWS (model/ukws.py) – final scoring/inference wrappers
– Utility layers (model/utils.py)
D. Losses & Metrics
– criterion/total.py (overall loss)
– criterion/utils.py (individual loss components: contrastive, classification)
E. Training Orchestrator
– train.py – CLI entry point that wires data loaders, model, losses, optimizer, TensorBoard logging
F. Environment
– docker/Dockerfile – reproducible container with PyTorch, CUDA, dependencies
G. Outputs & Monitoring
– Checkpoints (via train.py)
– TensorBoard logs (./log/)

##### Relationships & Data Flows
- Audio path: Data Source → Data Loader → Log-MelSpectrogram → Speech Embedding → Encoder → Extractor
- Text path: User-defined keyword → G2P module → Phoneme Embedding → Encoder → Extractor
- Extractor outputs fused representation → Discriminator → Loss computation → Backprop → Model update
- Training script orchestrates loops, writes TensorBoard logs, saves checkpoints

##### Key Architectural Patterns & Principles
- Two-branch (bimodal) feature extraction and late fusion
- Modular separation: data, preprocessing, model definition, losses, orchestration
- Containerized environment for reproducibility

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Container" 
        style DockerContainer fill:none,stroke:#888,stroke-width:2px
        subgraph "Data Sources & Loaders"
            LP["LibriPhrase Dataset"]:::data
            GSC["Google Speech Commands"]:::data
            QKS["Qualcomm Keyword Speech"]:::data
            TextInput["User Keywords\n(Text Input)"]:::data
            G2P["G2P Module"]:::preprocessing
        end

        subgraph "Preprocessing & Feature Extraction"
            LogMel["Log-Mel Spectrogram"]:::preprocessing
            TFHub["Speech Embeddings\nTF-Hub SavedModel"]:::external
            SpeechEmb["Speech Embedding Wrapper"]:::preprocessing
        end

        subgraph "Model Core"
            ENCA["Encoder\nmodel/encoder.py"]:::model
            EXT["Extractor\nmodel/extractor.py"]:::model
            DIS["Discriminator\nmodel/discriminator.py"]:::model
            UKWS["UKWS Wrapper\nmodel/ukws.py"]:::model
            Utils["Utility Layers\nmodel/utils.py"]:::model
        end

        subgraph "Loss & Training"
            LossUtils["Loss Components\ncriterion/utils.py"]:::training
            LossTotal["Total Loss\ncriterion/total.py"]:::training
            Train["Training Orchestrator\ntrain.py"]:::training
            TB["TensorBoard Logs\n./log/"]:::external
            CKPT["Model Checkpoints\ncheckpoints/"]:::external
        end
    end

    %% Data flows
    LP -->|audio| LogMel
    GSC -->|audio| LogMel
    QKS -->|audio| LogMel
    TextInput -->|text| G2P

    LogMel --> SpeechEmb
    SpeechEmb --> ENCA
    TFHub --> SpeechEmb

    G2P --> ENCA

    ENCA --> EXT
    EXT --> DIS
    DIS --> LossTotal
    LossUtils --> LossTotal

    LossTotal -->|"backprop"| ENCA
    LossTotal -->|"backprop"| EXT

    Train --> LossTotal
    Train --> ENCA
    Train --> EXT

    Train --> TB
    Train --> CKPT

    %% Click events
    click LP "https://github.com/ncsoft/phonmatchnet/blob/main/dataset/libriphrase.py"
    click GSC "https://github.com/ncsoft/phonmatchnet/blob/main/dataset/google.py"
    click QKS "https://github.com/ncsoft/phonmatchnet/blob/main/dataset/qualcomm.py"
    click G2P "https://github.com/ncsoft/phonmatchnet/tree/main/dataset/g2p/g2p_en/"
    click LogMel "https://github.com/ncsoft/phonmatchnet/blob/main/model/log_melspectrogram.py"
    click TFHub "https://github.com/ncsoft/phonmatchnet/tree/main/model/google_speech_embedding/"
    click SpeechEmb "https://github.com/ncsoft/phonmatchnet/blob/main/model/speech_embedding.py"
    click ENCA "https://github.com/ncsoft/phonmatchnet/blob/main/model/encoder.py"
    click EXT "https://github.com/ncsoft/phonmatchnet/blob/main/model/extractor.py"
    click DIS "https://github.com/ncsoft/phonmatchnet/blob/main/model/discriminator.py"
    click UKWS "https://github.com/ncsoft/phonmatchnet/blob/main/model/ukws.py"
    click Utils "https://github.com/ncsoft/phonmatchnet/blob/main/model/utils.py"
    click LossTotal "https://github.com/ncsoft/phonmatchnet/blob/main/criterion/total.py"
    click LossUtils "https://github.com/ncsoft/phonmatchnet/blob/main/criterion/utils.py"
    click Train "https://github.com/ncsoft/phonmatchnet/blob/main/train.py"
    click DockerContainer "https://github.com/ncsoft/phonmatchnet/tree/main/docker/Dockerfile"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#036,stroke-width:1px
    classDef preprocessing fill:#D0FFD8,stroke:#070,stroke-width:1px
    classDef model fill:#FFD0D0,stroke:#C00,stroke-width:1px
    classDef training fill:#FFE0B0,stroke:#C60,stroke-width:1px
    classDef external fill:#EEEEEE,stroke:#666,stroke-width:1px
```
