How gpapagiannis/r-plus-x-hand2actions works
Project type and purpose
• This is a research-oriented data-processing pipeline (not a web app or service) whose goal is to take human video demonstrations and extract corresponding robot gripper actions. It is split conceptually into:
• Retrieval phase – frame by frame extraction of hand keypoints and mapping them to gripper poses.
• (Referenced) Execution phase – zero-shot execution using Keypoint Action Tokens (not part of this repo).
File structure
• Top level:
• video_to_gripper_hamer_kpts.py → main orchestration script (Retrieval pipeline)
• util_functions.py → helper routines for point-cloud processing, coordinate transforms
• global_vars.py → configuration (paths, intrinsics, task toggles)
• renderer.py → patched renderer for HaMeR integration
• conda_env.yaml & requirements.txt → environment & dependencies (Python ≥3.10, PyTorch, NumPy, HaMeR, MANO)
• assets/
– grasp_fanta/, pick_up_phone/ ⇒ input demonstration data (depths.npy + rgbs.npy)
– utils/ ⇒ pretrained intrinsics matrices & gripper point-cloud templates
README for additional insights
• The README confirms:
• Two demo tasks (grasp_fanta, pick_up_phone) with different camera motions + intrinsics
• Dependency on HaMeR (hand tracking) and MANO (hand model)
• A dense gripper point-cloud mapped via hard-coded indices in the script
Main components & their relationships (to appear
A. Input Data
• Raw RGB frames + depth maps (assets/<task>/*.npy)
• Camera intrinsics (assets/utils/*.npy or global_vars.py)
B. Preprocessing
• Frame-by-frame loader in video_to_gripper_hamer_kpts.py
C. Hand Tracking (external service)
• HaMeR model → 3D hand mesh + keypoints
• Patched renderer.py is injected into HaMeR
D. Hand Model Processing
• MANO hand model (PyTorch) refines mesh/joint output
E. Point-cloud Generation & Merging
• util_functions: fuses depth+mesh → hand point cloud
F. Gripper Mapping
• Load gripper_point_cloud_dense.npy
• Heuristic mapping (index_front, thumb_front, wrist indices)
• Compute rigid transform from hand to gripper pose per frame
G. Pose Sequence Assembly
• Sequence of gripper poses → saved under scene_files/ per frame
• Export hand_joints_kpts_3d.npy for downstream training or execution
H. Visualization
• vizualizer.ipynb loads scene files and renders point clouds + meshes
Architectural patterns & design principles
• Pipeline (linear data-flow) architecture: each stage’s output is the next stage’s input
• Modularization:
• Configuration isolated in global_vars.py
• Helper functions in util_functions.py
• Main orchestration in a single script for clarity in research code
• External services are clearly demarcated (HaMeR, MANO)