How ncsoft/phonmatchnet works
Project Type and Purpose
• This is a research-oriented ML pipeline (PyTorch) for zero-shot keyword spotting guided by phonemes.
• It ingests audio and text (user-defined keywords), extracts features, fuses them in a neural model, and trains end-to-end.
High-Level Components
A. Data Sources
– LibriPhrase dataset
– Google Speech Commands
– Qualcomm Keyword Speech
B. Data-Loading & Preprocessing
– dataset/… modules (g2p for grapheme-to-phoneme, google.py, libriphrase.py, qualcomm.py)
– Feature extraction: model/log_melspectrogram.py
– Speech embeddings: model/google_speech_embedding (TF-Hub SavedModel) & model/speech_embedding.py
C. Model Core
– Extractor (model/extractor.py) – stacks and fuses audio/text features
– Encoder (model/encoder.py) – projection heads for each modality
– Discriminator (model/discriminator.py) – scoring network for keyword vs. background
– UKWS (model/ukws.py) – final scoring/inference wrappers
– Utility layers (model/utils.py)
D. Losses & Metrics
– criterion/total.py (overall loss)
– criterion/utils.py (individual loss components: contrastive, classification)
E. Training Orchestrator
– train.py – CLI entry point that wires data loaders, model, losses, optimizer, TensorBoard logging
F. Environment
– docker/Dockerfile – reproducible container with PyTorch, CUDA, dependencies
G. Outputs & Monitoring
– Checkpoints (via train.py)
– TensorBoard logs (./log/)
Relationships & Data Flows
• Audio path: Data Source → Data Loader → Log-MelSpectrogram → Speech Embedding → Encoder → Extractor
• Text path: User-defined keyword → G2P module → Phoneme Embedding → Encoder → Extractor
• Extractor outputs fused representation → Discriminator → Loss computation → Backprop → Model update
• Training script orchestrates loops, writes TensorBoard logs, saves checkpoints
Key Architectural Patterns & Principles
• Two-branch (bimodal) feature extraction and late fusion
• Modular separation: data, preprocessing, model definition, losses, orchestration
• Containerized environment for reproducibility