# sutwangyan/mska architecture

> Architecture diagram of the GitHub repository sutwangyan/mska, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sutwangyan/mska
- Repository: https://github.com/sutwangyan/mska

## Overview

##### Project Type and Purpose
- This is a research‐oriented machine‐learning project implementing “MSKA”: a Multi‐Stream Keypoint Attention Network for Sign Language Recognition (SLR) and Translation (SLT).
- It is not a web or full‐stack app but a modular PyTorch training and evaluation pipeline.

##### Top‐Level File Structure Analysis
- Python entry points:
- train.py – drives both SLR and SLT training/evaluation using a YAML config
- recognition.py, translation.py – wrappers for inference in SLR/SLT modes
- Core modules:
- datasets.py – dataset loading and preprocessing logic
- model.py – MSKA multi‐stream attention network definition
- optimizer.py – optimizer and learning‐rate scheduler setup
- metrics.py, sacrebleu.py, Rouge.py – evaluation metrics for recognition/translation
- Tokenizer.py – text tokenization for translation head
- vl_mapper.py, Visualhead.py – modules for visual‐language mapping and attention heads
- utils.py – shared utilities (logging, checkpointing, config handling)
- Configuration folder: configs/*.yaml – hyperparameters and dataset‐specific settings (SLR vs. SLT)
- Data folder: data/* – prepared keypoint pickles and raw data directories for each benchmark
- Supplementary scripts: phoenix_cleanup.py – data cleanup for Phoenix datasets

##### Key Insights from README
- Two modes:
a. MSKA‐SLR: output gloss sequences (WER evaluation)
b. MSKA‐SLT: output text translations (BLEU/Rouge evaluation)
- Pre‐extracted HRNet keypoints are input features
- External pretrained language models (mbart) initialize translation head
- Configurable via command line: `python train.py --config configs/<dataset>_s2{g,t}.yaml`

##### Components
A. Data Ingestion & Preprocessing
- Raw video/keypoint files (data/) → datasets.py
- phoenix_cleanup.py (optional)
- Keypoint input loader → gloss2ids.pkl mapping
B. Model Core (model.py)
- Multi‐Stream Keypoint Attention Module
– Streams: left hand, right hand, body keypoints
– Fusion strategies (keypoint fusion, head fusion)
- Two output heads:
– SLR head (sequence of gloss tokens)
– SLT head (translation decoder)
C. Tokenization & Language Models
- Tokenizer.py for input text/translation preprocessing
- Pretrained mbart models for SLT decoder initialization
D. Training Loop (train.py)
- Reads YAML config → builds model, optimizer (optimizer.py), datasets
- Forward pass → compute loss (recognition/translation)
- Backprop and parameter update → checkpointing
E. Evaluation & Metrics
- SLR: WER via metrics.py
- SLT: BLEU via sacrebleu.py, Rouge via Rouge.py
F. Utilities & Config
- utils.py for logging, config parsing, checkpoint management

##### Relationships & Data Flow
- Data Folder → Dataset Loader → Model Core
- Model Core →
- SLR Branch → Gloss Output → metrics.py → WER
- SLT Branch → Text Output → sacrebleu.py + Rouge.py → BLEU/Rouge
- Configs drive hyperparameters and dataset selection
- Optimizer module interfaces with model parameters
- Tokenizer and pretrained language models feed into SLT branch

##### Architectural Patterns & Design Principles
- Modular separation: data, model, optimizer, metrics, utilities
- Config‐driven experimentation (YAML files)
- Support for two distinct tasks (SLR vs. SLT) via shared backbone + separate heads
- Reusable evaluation modules for standard metrics

##### Detailed Elements
- File/Module names near each component (e.g., model.py, datasets.py)
- Config folder feeding into train.py
- Two output paths (SLR vs. SLT) with corresponding metric modules
- External artifacts: data/, pretrained_models/, HRNet keypoints

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Preprocessing"
        datasets["datasets.py"]:::data
        cleanup["phoenix_cleanup.py"]:::data
        D(("data/")):::data
    end

    subgraph "Model Core"
        model["model.py"]:::model
        vl["vl_mapper.py"]:::model
        vhead["Visualhead.py"]:::model
    end

    subgraph "Tokenization & Language Models"
        tok["Tokenizer.py"]:::model
        pretrained(("mbart pretrained")):::external
    end

    subgraph "Training Loop & Config"
        train["train.py"]:::logic
        opt["optimizer.py"]:::logic
        util["utils.py"]:::logic
        configs["configs/"]:::logic
    end

    subgraph "Inference Wrappers"
        rec["recognition.py"]:::logic
        trans["translation.py"]:::logic
    end

    subgraph "Evaluation & Metrics"
        met["metrics.py"]:::logic
        bleu["sacrebleu.py"]:::logic
        rouge["Rouge.py"]:::logic
    end

    D -->|"keypoint files"| datasets
    cleanup -->|"cleaned data"| datasets
    datasets -->|"dataset loader"| train
    configs -->|"config YAML"| train
    train -->|"builds optimizer"| opt
    opt -->|"optimizer config"| train
    train -->|"initialize model"| model
    pretrained -->|"initialize SLT head"| tok
    tok -->|"token embeddings"| model
    model -->|"gloss tokens"| slr["SLR head"]:::model
    model -->|"translations"| slt["SLT head"]:::model
    slr -->|"gloss tokens"| met
    slt -->|"translations"| bleu
    slt -->|"translations"| rouge

    click datasets "https://github.com/sutwangyan/mska/blob/main/datasets.py"
    click cleanup "https://github.com/sutwangyan/mska/blob/main/phoenix_cleanup.py"
    click D "https://github.com/sutwangyan/mska/tree/main/data/"
    click model "https://github.com/sutwangyan/mska/blob/main/model.py"
    click vl "https://github.com/sutwangyan/mska/blob/main/vl_mapper.py"
    click vhead "https://github.com/sutwangyan/mska/blob/main/Visualhead.py"
    click tok "https://github.com/sutwangyan/mska/blob/main/Tokenizer.py"
    click pretrained "https://github.com/sutwangyan/mska/tree/main/pretrained_models/"
    click train "https://github.com/sutwangyan/mska/blob/main/train.py"
    click opt "https://github.com/sutwangyan/mska/blob/main/optimizer.py"
    click util "https://github.com/sutwangyan/mska/blob/main/utils.py"
    click rec "https://github.com/sutwangyan/mska/blob/main/recognition.py"
    click trans "https://github.com/sutwangyan/mska/blob/main/translation.py"
    click met "https://github.com/sutwangyan/mska/blob/main/metrics.py"
    click bleu "https://github.com/sutwangyan/mska/blob/main/sacrebleu.py"
    click rouge "https://github.com/sutwangyan/mska/blob/main/Rouge.py"

    classDef data fill:#E3F2FD,stroke:#0288D1,color:#01579B
    classDef model fill:#C8E6C9,stroke:#2E7D32,color:#1B5E20
    classDef logic fill:#FFE0B2,stroke:#EF6C00,color:#E65100
    classDef external fill:#F0F4C3,stroke:#9E9D24,color:#827717
```
