How yuangongnd/gopt works
Project Type and Purpose
• Type: Open-source ML research tool / pipeline for multi-aspect, multi-granularity pronunciation assessment.
• Purpose: Given raw speech (or precomputed GOP features), preprocess into fixed-length sequences, feed into a Transformer-based model (GOPT), and evaluate pronunciation quality at phoneme, word, and utterance levels. Supports both local scripts and Google Colab execution.
File Structure
• Top-level dirs:
· colab: notebook orchestrating the pipeline on Google Colab
· data: stored GOP features and sequence datasets
· exp: experiment outputs (trained models, logs, predictions)
· figure: static assets for README visualization
· pretrained_models: saved GOPT checkpoints and conversion scripts
· src: core code, subdivided into
– extract_kaldi_gop: scripts for GOP feature extraction (requires Kaldi)
– prep_data: scripts to convert GOP features into sequence tensors
– models: baseline.py and gopt.py defining PyTorch models
– traintest.py and run.sh: entry point for training and evaluation
• Config & tooling:
· requirements.txt for Python deps
· steps_of_inference.md for end-user inference guide
README for Insights
• Data flows: raw Speechocean762 → (optional) Kaldi recipe → GOP features → seq_data generators → model training & evaluation
• Supports two execution modes:
a. Local (requires Kaldi for raw GOP extraction, or skip with downloaded data)
b. Colab (uses precomputed GOP, zero-setup)
• Outputs: experiment artifacts saved under exp_dir, pretrained checkpoints under pretrained_models.
Main Components
a. Data Ingestion
– Speechocean762 dataset (raw WAVs, transcripts)
– Precomputed GOP features (downloadable ZIP or Dropbox)
b. Feature Extraction
– extract_gop_feats.py / extract_gop_feats_word.py
c. Data Preparation
– gen_seq_data_{phn,word,utt}.py in src/prep_data
– Pads and masks sequences to fixed length
– Outputs: data/seq_data_*
d. Model Training & Evaluation
– shell script run.sh → traintest.py
– Loads sequence data → instantiates GOPT (models/gopt.py) or baseline
– Computes losses, metrics; logs to exp/<experiment_name>
e. Pretrained Models & Inference
– load_model.py for converting old logs
– steps_of_inference.md / Colab notebook invokes data prep and traintest stages in cloud
f. Experiment Artifacts
– exp: trained checkpoints, result.csv, logs
– pretrained_models: packaged best_audio_model.pth, result.csv
Relationships and Data Flow
1. Raw data → (optional) Kaldi GOP → GOP features
2. GOP features → seq_data generators → sequence datasets
3. seq_data → run.sh → traintest.py → model
4. Model outputs → metrics & checkpoints → exp
5. Inference path: raw audio → data/seq_data (via Colab) → load pretrained model → predictions
Architectural Patterns & Principles
• Pipeline pattern: distinct stages with clear hand-off artifacts in file system
• Modularity: each script or module has a single responsibility (extract, prep, train, evaluate)
• Extensibility: drop-in new model in src/models, update __init__.py and traintest.py
• Separation of concerns: data, preprocessing, modeling, evaluation, reporting
Technologies and External Dependencies
• Python 3, PyTorch
• Kaldi (for raw GOP extraction)
• Google Colab (optional execution environment)
• Shell scripting (run.sh), SLURM support for HPC
• JSON (scores.json), CSV (result.csv)