# rhss10/joint-apa-mdd-mtl architecture

> Architecture diagram of the GitHub repository rhss10/joint-apa-mdd-mtl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rhss10/joint-apa-mdd-mtl
- Repository: https://github.com/rhss10/joint-apa-mdd-mtl

## Overview

##### Project type and purpose
- This is an open-source machine-learning research codebase for “A Joint Model for Pronunciation Assessment and Mispronunciation Detection and Diagnosis with Multi-task Learning.”
- It implements a data-to-evaluation pipeline: preparing speech datasets, fine-tuning a self-supervised learning (SSL) model on a phone-recognition task, then training a joint multi-task model for automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD), and finally running tests/analysis.

##### File structure
- Top-level directories:
- auxiliary-phone-recognition – scripts to fine-tune/test the phone-recognition model (trainer_train.py, trainer_test.py).
- data – data preparation scripts (create_datasets.py, preprocess_datasets.py).
- multi-task-learning – the core joint training module (train.py) plus subdirectories models/ and test/.
- vocab – contains vocabulary JSON for tokenization/classification.
- Configuration & orchestrator: requirements.txt defines Python dependencies; README.md provides usage instructions.
- External/tool dependencies: uses Hugging Face Transformers, PyTorch (implied), Kaldi SCTK for MDD evaluation.

##### README insights
- Step 1: Prepare three speech corpora and convert to Hugging Face Datasets.
- Step 2: Auxiliary fine-tuning of Wav2Vec2 or HuBERT for phone recognition.
- Step 3: Joint training of APA+MDD model using the fine-tuned checkpoint or raw SSL model.
- Step 4: Evaluation and correlation analysis, invoking Kaldi SCTK utilities and custom scripts.

- Raw Data Store – audio files and transcriptions for TIMIT, L2-arctic, Speechocean762.
- Data Prep Module – create_datasets.py → Hugo Datasets; preprocess_datasets.py → tokenization/vocab.
- Vocabulary Service – vocab/vocab.json.
- Auxiliary Phone-Recognition Trainer – trainer_train.py (inputs: preprocessed data, base SSL model; outputs: fine-tuned checkpoint).
- Auxiliary Phone-Recognition Tester – trainer_test.py.
- Multi-Task Learning Trainer – multi-task-learning/train.py (inputs: checkpoint or raw SSL; outputs: joint model).
- Model Zoo – storage for trainer outputs and experiment logs.
- Evaluation & Analysis Module – multi-task-learning/test/test.sh, correlation.py, ins_del_sub_cor_analysis, wer ops scripts, Kaldi SCTK.
- External Tools – Kaldi SCTK, Hugging Face Hub (optional).
- Raw Data → Data Prep → Preprocessed Datasets + Vocabulary → Auxiliary Trainer → Fine-tuned Checkpoint → MTL Trainer → Joint Model → Evaluation Module → Correlation Reports.
- Evaluation Module ↔ Kaldi SCTK utilities.
- All training modules ← requirements.txt (dependency injection) and compute resources.
c. Architectural patterns & design principles:
- Pipeline/ETL pattern for data preparation.
- Modular separation: distinct repos for auxiliary and core training flows.
- Reuse of SSL backbone (Wav2Vec2/HuBERT) with fine-tuning.
- Multi-task learning head (joint_apa_mdd.py in models/).
d. Technologies & frameworks:
- Python 3.8, PyTorch, Hugging Face Transformers & Datasets
- Kaldi SCTK for MDD evaluation
- Shell scripts for orchestration (test.sh, train.sh)
- Vocabulary JSON for CTC/classification heads

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData[(Raw Data Store)]:::data
        CreateDatasets["create_datasets.py"]:::data
        PreprocessDatasets["preprocess_datasets.py"]:::data
        VocabJson["vocab.json"]:::data
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        AuxTrainer["trainer_train.py"]:::training
        AuxTester["trainer_test.py"]:::training
        MTLTrainer["train.py"]:::training
        JointModel["joint_apa_mdd.py"]:::training
    end

    %% Model Storage
    ModelZoo[(Model Zoo)]:::data

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        TestSh["test.sh"]:::evaluation
        Correlation["correlation.py"]:::evaluation
        InsDelSub["ins_del_sub_cor_analysis.py"]:::evaluation
        KaldiAlign["kaldi-align.sh"]:::evaluation
        TestPy["test.py"]:::evaluation
        WerOps["wer_ops_details.pl"]:::evaluation
        WerPerUtt["wer_per_utt_details.pl"]:::evaluation
    end

    %% External Tools
    subgraph "External Tools"
        direction TB
        KaldiSCTK[(Kaldi SCTK)]:::external
        HFHub[(Hugging Face Hub)]:::external
    end

    %% Config Files
    ConfigReq["requirements.txt"]:::config
    ConfigReadme["README.md"]:::config

    %% Connections
    RawData --> CreateDatasets --> PreprocessDatasets --> VocabJson
    PreprocessDatasets --> AuxTrainer
    AuxTrainer --> AuxTester
    AuxTrainer --> ModelZoo
    AuxTester --> ModelZoo
    ModelZoo --> MTLTrainer
    VocabJson --> MTLTrainer
    MTLTrainer --> JointModel
    JointModel --> ModelZoo
    JointModel --> TestSh
    TestSh --> Correlation --> InsDelSub --> WerPerUtt
    Correlation --> WerOps
    TestSh --> KaldiAlign
    KaldiAlign --> KaldiSCTK
    Correlation --> KaldiSCTK
    TestSh --> TestPy
    MTLTrainer --> ConfigReq
    AuxTrainer --> ConfigReq
    PreprocessDatasets --> ConfigReq
    CreateDatasets --> ConfigReq
    ConfigReq --> HFHub

    %% Click Events
    click CreateDatasets "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/data/create_datasets.py"
    click PreprocessDatasets "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/data/preprocess_datasets.py"
    click VocabJson "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/vocab/vocab.json"
    click AuxTrainer "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/auxiliary-phone-recognition/trainer_train.py"
    click AuxTester "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/auxiliary-phone-recognition/trainer_test.py"
    click MTLTrainer "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/train.py"
    click JointModel "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/models/joint_apa_mdd.py"
    click TestSh "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/test.sh"
    click Correlation "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/correlation.py"
    click InsDelSub "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/ins_del_sub_cor_analysis.py"
    click KaldiAlign "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/kaldi-align.sh"
    click TestPy "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/test.py"
    click WerOps "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/utils/wer_ops_details.pl"
    click WerPerUtt "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/multi-task-learning/test/utils/wer_per_utt_details.pl"
    click ConfigReq "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/requirements.txt"
    click ConfigReadme "https://github.com/rhss10/joint-apa-mdd-mtl/blob/master/README.md"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085;
    classDef training fill:#d4edda,stroke:#155724,color:#155724;
    classDef evaluation fill:#fff3cd,stroke:#856404,color:#856404;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef config fill:#f8f9fa,stroke:#343a40,color:#343a40;
```
