How rhss10/joint-apa-mdd-mtl works
Project type and purpose
• This is an open-source machine-learning research codebase for “A Joint Model for Pronunciation Assessment and Mispronunciation Detection and Diagnosis with Multi-task Learning.”
• It implements a data-to-evaluation pipeline: preparing speech datasets, fine-tuning a self-supervised learning (SSL) model on a phone-recognition task, then training a joint multi-task model for automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD), and finally running tests/analysis.
File structure
• Top-level directories:
• auxiliary-phone-recognition – scripts to fine-tune/test the phone-recognition model (trainer_train.py, trainer_test.py).
• data – data preparation scripts (create_datasets.py, preprocess_datasets.py).
• multi-task-learning – the core joint training module (train.py) plus subdirectories models/ and test/.
• vocab – contains vocabulary JSON for tokenization/classification.
• Configuration & orchestrator: requirements.txt defines Python dependencies; README.md provides usage instructions.
• External/tool dependencies: uses Hugging Face Transformers, PyTorch (implied), Kaldi SCTK for MDD evaluation.
README insights
• Step 1: Prepare three speech corpora and convert to Hugging Face Datasets.
• Step 2: Auxiliary fine-tuning of Wav2Vec2 or HuBERT for phone recognition.
• Step 3: Joint training of APA+MDD model using the fine-tuned checkpoint or raw SSL model.
• Step 4: Evaluation and correlation analysis, invoking Kaldi SCTK utilities and custom scripts.
• Raw Data Store – audio files and transcriptions for TIMIT, L2-arctic, Speechocean762.
• Data Prep Module – create_datasets.py → Hugo Datasets; preprocess_datasets.py → tokenization/vocab.
• Vocabulary Service – vocab/vocab.json.
• Auxiliary Phone-Recognition Trainer – trainer_train.py (inputs: preprocessed data, base SSL model; outputs: fine-tuned checkpoint).
• Auxiliary Phone-Recognition Tester – trainer_test.py.
• Multi-Task Learning Trainer – multi-task-learning/train.py (inputs: checkpoint or raw SSL; outputs: joint model).
• Model Zoo – storage for trainer outputs and experiment logs.
• Evaluation & Analysis Module – multi-task-learning/test/test.sh, correlation.py, ins_del_sub_cor_analysis, wer ops scripts, Kaldi SCTK.
• External Tools – Kaldi SCTK, Hugging Face Hub (optional).
• Raw Data → Data Prep → Preprocessed Datasets + Vocabulary → Auxiliary Trainer → Fine-tuned Checkpoint → MTL Trainer → Joint Model → Evaluation Module → Correlation Reports.
• Evaluation Module ↔ Kaldi SCTK utilities.
• All training modules ← requirements.txt (dependency injection) and compute resources.
c. Architectural patterns & design principles:
• Pipeline/ETL pattern for data preparation.
• Modular separation: distinct repos for auxiliary and core training flows.
• Reuse of SSL backbone (Wav2Vec2/HuBERT) with fine-tuning.
• Multi-task learning head (joint_apa_mdd.py in models/).
d. Technologies & frameworks:
• Python 3.8, PyTorch, Hugging Face Transformers & Datasets
• Kaldi SCTK for MDD evaluation
• Shell scripts for orchestration (test.sh, train.sh)
• Vocabulary JSON for CTC/classification heads