# henrylittle/glofe architecture

> Architecture diagram of the GitHub repository henrylittle/glofe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/henrylittle/glofe
- Repository: https://github.com/henrylittle/glofe

## Overview

##### Project type and purpose
- This is a research-oriented, end-to-end sign-language translation pipeline (GloFE) implemented in Python/PyTorch, designed to go from raw video → pose features → text translation.
- It includes data-preparation (pose extraction), model definitions (pose backbone, graph modules, transformer encoder–decoder), training/inference scripts, notebooks for analysis, and utilities.

##### File structure
- Top-level directories:
- models/       ← contains all neural modules (pose backbones, graph networks, transformer variants, losses)
- scripts/      ← dataset-specific training-and-testing shell wrappers (how2sign/, openasl/)
- tools/        ← pose‐extraction helper scripts (MMPose/OpenPose orchestration)
- notebooks/    ← exploratory analysis, tokenizer training, evaluation
- utils/        ← shared helper code (beam search, config parsing)
- Top-level files: train_openasl_pose_DDP_inter_VN.py & train_how2_pose_DDP_inter_VN.py orchestrate DDP training.
- Config/build/deployment indicators: requirements.txt, conda instructions in README, GPU scripts in tools and scripts.

##### Key insights from README
- Two datasets: How2Sign and OpenASL, each requiring different pose extraction (OpenPose vs. MMPose).
- Workflow:
a) Extract pose sequences → b) Tokenizer training or load pretrained → c) Model training (single-GPU or DDP) → d) Inference / evaluation (BLEU, BLEURT).
- Dependencies: PyTorch, torchvision, MMPose, OpenPose, mmcv, etc.
- External services: none (all local), but depends on Google Drive for pretrained weights.

1) Data Sources
- Raw videos from How2Sign / OpenASL
2) Pose Extraction Layer
- OpenPose pipeline (How2Sign)
- MMPose pipeline (OpenASL)
→ Outputs: JSON / .pkl keypoint sequences
3) Feature & Token Preparation
- Pose feature loader (models/pose_backbones.py)
- Tokenizer (BPE models stored in notebooks/)
4) Model Core (models/)
- Graph Network modules (models/graph/*.py) – spatial‐temporal graph conv
- Embedding & positional encodings (models/embedding.py)
- Transformer encoder (models/trans_model*.py)
- Decoder with inter‐modal contrastive loss (models/inter_vn_loss.py, vn_loss.py)
- Loss functions (models/loss.py)
5) Training Orchestration
- Single-GPU scripts: train_*.py
- DDP scripts: scripts/*/train_ddp.sh
- Logging & checkpointing (models/ckpt/, exp_config.json)
6) Inference & Evaluation
- scripts/*/test.sh
- Beam search (utils/beam_search.py) → predicted text → BLEU/BLEURT metrics
7) Analysis & Visualization
- notebooks/ for word embedding analysis, metric computation

Raw Video → Pose Extraction → Pose Feature Files → DataLoader → Model Core → Loss/Optimization → Checkpoints
Checkpoints + Tokenizer → Inference Script → Predictions → Metrics → Notebooks for analysis

c. Architectural patterns & principles:
- Modular pipeline: separate extraction, preprocessing, modeling, evaluation
- Pluggable pose backbones & graph modules (easily swap MMPose vs OpenPose)
- Transformer encoder–decoder paradigm
- Distributed Data-Parallel training (torchrun)
- Reusable utilities and shell wrappers for reproducibility

d. Key technologies:
- Python 3.8, PyTorch 1.10
- MMPose / mmcv / mmdet
- OpenPose
- Transformers (custom implementation in models/trans_model*.py)
- DistributedDataParallel (torch.distributed)
- BPE tokenizers (tokenizers.json)

##### Final advice
- Be exhaustive: break the “Model Core” into each submodule (embedding, graph, transformer, loss).
- Scripts and tooling as first-class components (don’t only show the Python code).
- Every data artifact (.json, .pkl, .pt) and process (train/test) is represented to reflect actual workflow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        RH2["Raw Videos: How2Sign"]:::dataio
        ROA["Raw Videos: OpenASL"]:::dataio
    end

    %% Pose Extraction Layer
    subgraph "Pose Extraction Layer"
        OP["OpenPose Extractor"]:::preproc
        subgraph "OpenASL Extraction"
            MPsh["extract_openasl.sh"]:::preproc
            MPmp["extract_openasl_mp.py"]:::preproc
            MPmiss["extract_openasl_mp_miss.py"]:::preproc
        end
    end

    RH2 -->|video files| OP
    ROA -->|video files| MPsh
    ROA -->|video files| MPmp
    ROA -->|video files| MPmiss

    click OP "https://github.com/henrylittle/glofe/blob/main/tools/extract.sh"
    click MPsh "https://github.com/henrylittle/glofe/blob/main/tools/extract_openasl.sh"
    click MPmp "https://github.com/henrylittle/glofe/blob/main/tools/extract_openasl_mp.py"
    click MPmiss "https://github.com/henrylittle/glofe/blob/main/tools/extract_openasl_mp_miss.py"

    %% Pose Features Output
    PF["Pose Features (.json/.pkl)"]:::dataio
    OP -->|.json/.pkl keypoints| PF
    MPsh -->|.json/.pkl keypoints| PF
    MPmp -->|.json/.pkl keypoints| PF
    MPmiss -->|.json/.pkl keypoints| PF

    %% Preprocessing
    subgraph "Feature & Token Preparation"
        PFL["Pose Feature Loader"]:::preproc
        HTok["How2Sign Tokenizer"]:::preproc
        OTok["OpenASL Tokenizer"]:::preproc
    end

    PF -->|keypoint sequences| PFL
    PFL -->|"tensor inputs"| ModelCore

    click PFL "https://github.com/henrylittle/glofe/blob/main/models/pose_backbones.py"
    click HTok "https://github.com/henrylittle/glofe/blob/main/notebooks/how2sign/how2sign-bpe25000-tokenizer-uncased/tokenizer.json"
    click OTok "https://github.com/henrylittle/glofe/blob/main/notebooks/openasl-v1.0/openasl-bpe25000-tokenizer-uncased/tokenizer.json"

    %% Model Core
    subgraph "Model Core" 
        subgraph "Embedding Modules"
            EMB["Embedding & Positional Encodings"]:::model
        end
        subgraph "Graph Network Modules"
            GM1["mmpose_p76"]:::model
            GM2["openpose_78"]:::model
            GM3["openpose_p76"]:::model
            GM4["tools"]:::model
        end
        subgraph "Base Graph Models"
            GMBase1["ctrgcn_base"]:::model
            GMBase2["ctrgcn_base_p76"]:::model
        end
        subgraph "Transformer Encoder–Decoder"
            TM1["trans_model"]:::model
            TM2["trans_model_inter_vn"]:::model
            TM3["trans_model_vn"]:::model
        end
        subgraph "Loss Modules"
            LV1["inter_vn_loss"]:::model
            LV2["vn_loss"]:::model
            LV3["loss"]:::model
        end
    end

    PFL --> EMB
    EMB --> GM1
    EMB --> GM2
    EMB --> GM3
    EMB --> GM4
    GM1 --> GMBase1
    GM2 --> GMBase1
    GM3 --> GMBase2
    GM4 --> GMBase2
    GMBase1 --> TM1
    GMBase1 --> TM2
    GMBase2 --> TM2
    GMBase2 --> TM3
    TM1 --> LV1
    TM2 --> LV1
    TM3 --> LV2
    LV1 --> LV3
    LV2 --> LV3

    click EMB "https://github.com/henrylittle/glofe/blob/main/models/embedding.py"
    click GM1 "https://github.com/henrylittle/glofe/blob/main/models/graph/mmpose_p76.py"
    click GM2 "https://github.com/henrylittle/glofe/blob/main/models/graph/openpose_78.py"
    click GM3 "https://github.com/henrylittle/glofe/blob/main/models/graph/openpose_p76.py"
    click GM4 "https://github.com/henrylittle/glofe/blob/main/models/graph/tools.py"
    click GMBase1 "https://github.com/henrylittle/glofe/blob/main/models/ctrgcn_base.py"
    click GMBase2 "https://github.com/henrylittle/glofe/blob/main/models/ctrgcn_base_p76.py"
    click TM1 "https://github.com/henrylittle/glofe/blob/main/models/trans_model.py"
    click TM2 "https://github.com/henrylittle/glofe/blob/main/models/trans_model_inter_vn.py"
    click TM3 "https://github.com/henrylittle/glofe/blob/main/models/trans_model_vn.py"
    click LV1 "https://github.com/henrylittle/glofe/blob/main/models/inter_vn_loss.py"
    click LV2 "https://github.com/henrylittle/glofe/blob/main/models/vn_loss.py"
    click LV3 "https://github.com/henrylittle/glofe/blob/main/models/loss.py"

    %% Training Orchestration
    subgraph "Training Orchestration"
        SGPU["Single-GPU Training Scripts"]:::training
        DDP["DDP Training Scripts"]:::training
        CKP["Checkpoint Storage"]:::training
    end

    LV3 --> SGPU
    LV3 --> DDP
    SGPU --> CKP
    DDP --> CKP

    click SGPU "https://github.com/henrylittle/glofe/blob/main/train_how2_pose_DDP_inter_VN.py"
    click SGPU "https://github.com/henrylittle/glofe/blob/main/train_openasl_pose_DDP_inter_VN.py"
    click DDP "https://github.com/henrylittle/glofe/blob/main/scripts/how2sign/train.sh"
    click DDP "https://github.com/henrylittle/glofe/blob/main/scripts/openasl/train.sh"
    click DDP "https://github.com/henrylittle/glofe/blob/main/scripts/how2sign/train_ddp.sh"
    click DDP "https://github.com/henrylittle/glofe/blob/main/scripts/openasl/train_ddp.sh"
    click CKP "https://github.com/henrylittle/glofe/blob/main/models/ckpt/ctr_76_e1/runs-88-100144.pt"
    click CKP "https://github.com/henrylittle/glofe/blob/main/models/ckpt/ctr_op78_mix_HF05_F64_e1/runs-82-93316.pt"

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        TESTH2["How2Sign Test Script"]:::evaluation
        TESTOA["OpenASL Test Scripts"]:::evaluation
        BS["Beam Search Utility"]:::evaluation
        MET["Metrics (BLEU/BLEURT)"]:::evaluation
    end

    CKP --> TESTH2
    CKP --> TESTOA
    TESTH2 --> BS
    TESTOA --> BS
    BS --> MET

    click TESTH2 "https://github.com/henrylittle/glofe/blob/main/scripts/how2sign/test.sh"
    click TESTOA "https://github.com/henrylittle/glofe/blob/main/scripts/openasl/test.sh"
    click TESTOA "https://github.com/henrylittle/glofe/blob/main/scripts/openasl/test_n.sh"
    click BS "https://github.com/henrylittle/glofe/blob/main/utils/beam_search.py"

    %% Analysis & Visualization
    subgraph "Analysis & Visualization"
        NA1["analyse_word_embd_how2sign.ipynb"]:::analysis
        NA2["analyse_word_embd_openasl.ipynb"]:::analysis
        NA3["rouge_bleurt.ipynb"]:::analysis
        NA4["train_toknizer_how2.ipynb"]:::analysis
        NA5["train_toknizer_openasl.ipynb"]:::analysis
    end

    MET --> NA1
    MET --> NA2
    MET --> NA3
    HTok --> NA4
    OTok --> NA5

    click NA1 "https://github.com/henrylittle/glofe/blob/main/notebooks/analyse_word_embd_how2sign.ipynb"
    click NA2 "https://github.com/henrylittle/glofe/blob/main/notebooks/analyse_word_embd_openasl.ipynb"
    click NA3 "https://github.com/henrylittle/glofe/blob/main/notebooks/rouge_bleurt.ipynb"
    click NA4 "https://github.com/henrylittle/glofe/blob/main/notebooks/train_toknizer_how2.ipynb"
    click NA5 "https://github.com/henrylittle/glofe/blob/main/notebooks/train_toknizer_openasl.ipynb"

    %% Styles
    classDef dataio fill:#c6f5c6,stroke:#2d7f2d,color:#000
    classDef preproc fill:#c6d9f5,stroke:#2a4f7f,color:#000
    classDef model fill:#f5d2a6,stroke:#7f4f2a,color:#000
    classDef training fill:#e2c6f5,stroke:#5f2d7f,color:#000
    classDef evaluation fill:#d0d0d0,stroke:#707070,color:#000
    classDef analysis fill:#f5e3c6,stroke:#7f5a2d,color:#000
```
