# gpapagiannis/r-plus-x-hand2actions architecture

> Architecture diagram of the GitHub repository gpapagiannis/r-plus-x-hand2actions, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gpapagiannis/r-plus-x-hand2actions
- Repository: https://github.com/gpapagiannis/r-plus-x-hand2actions

## Overview

##### Project type and purpose
- This is a research-oriented data-processing pipeline (not a web app or service) whose goal is to take human video demonstrations and extract corresponding robot gripper actions. It is split conceptually into:
- Retrieval phase – frame by frame extraction of hand keypoints and mapping them to gripper poses.
- (Referenced) Execution phase – zero-shot execution using Keypoint Action Tokens (not part of this repo).

##### File structure
- Top level:
- video_to_gripper_hamer_kpts.py → main orchestration script (Retrieval pipeline)
- util_functions.py → helper routines for point-cloud processing, coordinate transforms
- global_vars.py → configuration (paths, intrinsics, task toggles)
- renderer.py → patched renderer for HaMeR integration
- conda_env.yaml & requirements.txt → environment & dependencies (Python ≥3.10, PyTorch, NumPy, HaMeR, MANO)
- assets/
– grasp_fanta/, pick_up_phone/ ⇒ input demonstration data (depths.npy + rgbs.npy)
– utils/ ⇒ pretrained intrinsics matrices & gripper point-cloud templates

##### README for additional insights
- The README confirms:
- Two demo tasks (grasp_fanta, pick_up_phone) with different camera motions + intrinsics
- Dependency on HaMeR (hand tracking) and MANO (hand model)
- A dense gripper point-cloud mapped via hard-coded indices in the script

##### Main components & their relationships (to appear
A. Input Data
- Raw RGB frames + depth maps (assets/<task>/*.npy)
- Camera intrinsics (assets/utils/*.npy or global_vars.py)
B. Preprocessing
- Frame-by-frame loader in video_to_gripper_hamer_kpts.py
C. Hand Tracking (external service)
- HaMeR model → 3D hand mesh + keypoints
- Patched renderer.py is injected into HaMeR
D. Hand Model Processing
- MANO hand model (PyTorch) refines mesh/joint output
E. Point-cloud Generation & Merging
- util_functions: fuses depth+mesh → hand point cloud
F. Gripper Mapping
- Load gripper_point_cloud_dense.npy
- Heuristic mapping (index_front, thumb_front, wrist indices)
- Compute rigid transform from hand to gripper pose per frame
G. Pose Sequence Assembly
- Sequence of gripper poses → saved under scene_files/ per frame
- Export hand_joints_kpts_3d.npy for downstream training or execution
H. Visualization
- vizualizer.ipynb loads scene files and renders point clouds + meshes

##### Architectural patterns & design principles
- Pipeline (linear data-flow) architecture: each stage’s output is the next stage’s input
- Modularization:
- Configuration isolated in global_vars.py
- Helper functions in util_functions.py
- Main orchestration in a single script for clarity in research code
- External services are clearly demarcated (HaMeR, MANO)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw Data Stores
    GRASP["Grasp_fanta (*.npy)"]:::raw
    PICK["Pick_up_phone (*.npy)"]:::raw
    CAM["Camera Intrinsics & Templates"]:::raw

    %% Processing Pipeline
    Preprocess["Frame Loader & Preprocessor"]:::process
    Renderer["Patched Renderer"]:::external
    HaMeR["HaMeR Hand Tracker"]:::external
    MANO["MANO Hand Model"]:::external
    PointCloud["Point-Cloud Generator"]:::process
    Gripper["Gripper Mapper"]:::process
    SceneExp["Scene File Exporter"]:::output
    Viz["Visualization Notebook"]:::notebook

    %% Configuration Artifacts
    subgraph "Configuration"
        GV["global_vars.py"]:::config
        Conda["conda_env.yaml"]:::config
        Req["requirements.txt"]:::config
    end

    %% Data Flow
    GRASP --> Preprocess
    PICK --> Preprocess
    CAM --> Preprocess
    Preprocess --> Renderer
    Renderer --> HaMeR
    HaMeR --> MANO
    MANO --> PointCloud
    PointCloud --> Gripper
    Gripper --> SceneExp
    SceneExp --> Viz

    %% Config Dependencies
    GV -.-> Preprocess
    Conda -.-> Preprocess
    Req -.-> Preprocess

    %% Click Events
    click GRASP "https://github.com/gpapagiannis/r-plus-x-hand2actions/tree/main/assets/grasp_fanta/"
    click PICK "https://github.com/gpapagiannis/r-plus-x-hand2actions/tree/main/assets/pick_up_phone/"
    click CAM "https://github.com/gpapagiannis/r-plus-x-hand2actions/tree/main/assets/utils/"
    click Preprocess "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/video_to_gripper_hamer_kpts.py"
    click Renderer "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/renderer.py"
    click PointCloud "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/util_functions.py"
    click Gripper "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/assets/utils/gripper_point_cloud_dense.npy"
    click SceneExp "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/video_to_gripper_hamer_kpts.py"
    click Viz "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/vizualizer.ipynb"
    click GV "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/global_vars.py"
    click Conda "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/conda_env.yaml"
    click Req "https://github.com/gpapagiannis/r-plus-x-hand2actions/blob/main/requirements.txt"

    %% Styles
    classDef raw fill:#BBDEFB,stroke:#2196F3,color:#0D47A1
    classDef process fill:#C8E6C9,stroke:#388E3C,color:#1B5E20
    classDef external fill:#FFE0B2,stroke:#F57C00,color:#E65100
    classDef config fill:#E1BEE7,stroke:#8E24AA,color:#6A1B9A
    classDef output fill:#B2EBF2,stroke:#0097A7,color:#004D40
    classDef notebook fill:#F8BBD0,stroke:#C2185B,color:#880E4F
```
