How sutwangyan/mska works
Project Type and Purpose
• This is a research‐oriented machine‐learning project implementing “MSKA”: a Multi‐Stream Keypoint Attention Network for Sign Language Recognition (SLR) and Translation (SLT).
• It is not a web or full‐stack app but a modular PyTorch training and evaluation pipeline.
Top‐Level File Structure Analysis
• Python entry points:
• train.py – drives both SLR and SLT training/evaluation using a YAML config
• recognition.py, translation.py – wrappers for inference in SLR/SLT modes
• Core modules:
• datasets.py – dataset loading and preprocessing logic
• model.py – MSKA multi‐stream attention network definition
• optimizer.py – optimizer and learning‐rate scheduler setup
• metrics.py, sacrebleu.py, Rouge.py – evaluation metrics for recognition/translation
• Tokenizer.py – text tokenization for translation head
• vl_mapper.py, Visualhead.py – modules for visual‐language mapping and attention heads
• utils.py – shared utilities (logging, checkpointing, config handling)
• Configuration folder: configs/*.yaml – hyperparameters and dataset‐specific settings (SLR vs. SLT)
• Data folder: data/* – prepared keypoint pickles and raw data directories for each benchmark
• Supplementary scripts: phoenix_cleanup.py – data cleanup for Phoenix datasets
Key Insights from README
• Two modes:
a. MSKA‐SLR: output gloss sequences (WER evaluation)
b. MSKA‐SLT: output text translations (BLEU/Rouge evaluation)
• Pre‐extracted HRNet keypoints are input features
• External pretrained language models (mbart) initialize translation head
• Configurable via command line: python train.py --config configs/<dataset>_s2{g,t}.yaml
Components
A. Data Ingestion & Preprocessing
• Raw video/keypoint files (data/) → datasets.py
• phoenix_cleanup.py (optional)
• Keypoint input loader → gloss2ids.pkl mapping
B. Model Core (model.py)
• Multi‐Stream Keypoint Attention Module
– Streams: left hand, right hand, body keypoints
– Fusion strategies (keypoint fusion, head fusion)
• Two output heads:
– SLR head (sequence of gloss tokens)
– SLT head (translation decoder)
C. Tokenization & Language Models
• Tokenizer.py for input text/translation preprocessing
• Pretrained mbart models for SLT decoder initialization
D. Training Loop (train.py)
• Reads YAML config → builds model, optimizer (optimizer.py), datasets
• Forward pass → compute loss (recognition/translation)
• Backprop and parameter update → checkpointing
E. Evaluation & Metrics
• SLR: WER via metrics.py
• SLT: BLEU via sacrebleu.py, Rouge via Rouge.py
F. Utilities & Config
• utils.py for logging, config parsing, checkpoint management
Relationships & Data Flow
• Data Folder → Dataset Loader → Model Core
• Model Core →
• SLR Branch → Gloss Output → metrics.py → WER
• SLT Branch → Text Output → sacrebleu.py + Rouge.py → BLEU/Rouge
• Configs drive hyperparameters and dataset selection
• Optimizer module interfaces with model parameters
• Tokenizer and pretrained language models feed into SLT branch
Architectural Patterns & Design Principles
• Modular separation: data, model, optimizer, metrics, utilities
• Config‐driven experimentation (YAML files)
• Support for two distinct tasks (SLR vs. SLT) via shared backbone + separate heads
• Reusable evaluation modules for standard metrics
Detailed Elements
• File/Module names near each component (e.g., model.py, datasets.py)
• Config folder feeding into train.py
• Two output paths (SLR vs. SLT) with corresponding metric modules
• External artifacts: data/, pretrained_models/, HRNet keypoints