How henrylittle/glofe works
Project type and purpose
• This is a research-oriented, end-to-end sign-language translation pipeline (GloFE) implemented in Python/PyTorch, designed to go from raw video → pose features → text translation.
• It includes data-preparation (pose extraction), model definitions (pose backbone, graph modules, transformer encoder–decoder), training/inference scripts, notebooks for analysis, and utilities.
File structure
• Top-level directories:
• models/ ← contains all neural modules (pose backbones, graph networks, transformer variants, losses)
• scripts/ ← dataset-specific training-and-testing shell wrappers (how2sign/, openasl/)
• tools/ ← pose‐extraction helper scripts (MMPose/OpenPose orchestration)
• notebooks/ ← exploratory analysis, tokenizer training, evaluation
• utils/ ← shared helper code (beam search, config parsing)
• Top-level files: train_openasl_pose_DDP_inter_VN.py & train_how2_pose_DDP_inter_VN.py orchestrate DDP training.
• Config/build/deployment indicators: requirements.txt, conda instructions in README, GPU scripts in tools and scripts.
Key insights from README
• Two datasets: How2Sign and OpenASL, each requiring different pose extraction (OpenPose vs. MMPose).
• Workflow:
a) Extract pose sequences → b) Tokenizer training or load pretrained → c) Model training (single-GPU or DDP) → d) Inference / evaluation (BLEU, BLEURT).
• Dependencies: PyTorch, torchvision, MMPose, OpenPose, mmcv, etc.
• External services: none (all local), but depends on Google Drive for pretrained weights.
1) Data Sources
• Raw videos from How2Sign / OpenASL
2) Pose Extraction Layer
• OpenPose pipeline (How2Sign)
• MMPose pipeline (OpenASL)
→ Outputs: JSON / .pkl keypoint sequences
3) Feature & Token Preparation
• Pose feature loader (models/pose_backbones.py)
• Tokenizer (BPE models stored in notebooks/)
4) Model Core (models/)
• Graph Network modules (models/graph/*.py) – spatial‐temporal graph conv
• Embedding & positional encodings (models/embedding.py)
• Transformer encoder (models/trans_model*.py)
• Decoder with inter‐modal contrastive loss (models/inter_vn_loss.py, vn_loss.py)
• Loss functions (models/loss.py)
5) Training Orchestration
• Single-GPU scripts: train_*.py
• DDP scripts: scripts/*/train_ddp.sh
• Logging & checkpointing (models/ckpt/, exp_config.json)
6) Inference & Evaluation
• scripts/*/test.sh
• Beam search (utils/beam_search.py) → predicted text → BLEU/BLEURT metrics
7) Analysis & Visualization
• notebooks/ for word embedding analysis, metric computation
Raw Video → Pose Extraction → Pose Feature Files → DataLoader → Model Core → Loss/Optimization → Checkpoints
Checkpoints + Tokenizer → Inference Script → Predictions → Metrics → Notebooks for analysis
c. Architectural patterns & principles:
• Modular pipeline: separate extraction, preprocessing, modeling, evaluation
• Pluggable pose backbones & graph modules (easily swap MMPose vs OpenPose)
• Transformer encoder–decoder paradigm
• Distributed Data-Parallel training (torchrun)
• Reusable utilities and shell wrappers for reproducibility
d. Key technologies:
• Python 3.8, PyTorch 1.10
• MMPose / mmcv / mmdet
• OpenPose
• Transformers (custom implementation in models/trans_model*.py)
• DistributedDataParallel (torch.distributed)
• BPE tokenizers (tokenizers.json)
Final advice
• Be exhaustive: break the “Model Core” into each submodule (embedding, graph, transformer, loss).
• Scripts and tooling as first-class components (don’t only show the Python code).
• Every data artifact (.json, .pkl, .pt) and process (train/test) is represented to reflect actual workflow.