How zijian-wu/egohandkit works
EgoHandKit turns an image folder or video into per-frame hand detections, cleaned tracks, backend-specific MANO mesh estimates, and—by default—Omega camera and world-space hand results with visualizations. run.py coordinates the four documented passes; mesh_recovery.py and the backend runners handle mesh inference, while the Omega runtime derives world results. Optional ViTPose-assisted detection and configurable HMR backends are shown as branches. The README documents these stages and cache behavior; sampled source confirms sequence batching and Omega visualization, but not every orchestration call, so uncertain internal wiring is omitted.
Main entry points: runner.py Omega runtime · run.py Pipeline CLI · mesh_recovery.py Mesh orchestration · bbox_utils.py Hand detection
Components 25
Input and detection
- Pipeline CLI
run.py - Hand detection
bbox_utils.py - YOLO detector
- ViTPose merge
vitpose_model.py - Track cleanup
bbox_utils.py - Detection cache
- Cleaned tracks
Mesh recovery
- Mesh orchestration
mesh_recovery.py - Runner factory
hmr_backends/runners/factory.py - Batch inference
hmr_backends/runners/batch_runner.py - HaWoR inference
hmr_backends/runners/hawor_runner.py - Sequence crops
hmr_backends/datasets/vitdet_dataset.py - HMR backends
hmr_backends/models/hamer.py - MANO head
hmr_backends/models/heads/mano_head.py - Frame mesh results
Omega world derivation
- Omega runtime
hmr_backends/omega/runner.py - VGGT-Omega model
hmr_backends/omega/models/vggt_omega.py - Camera recovery
hmr_backends/omega/models/heads/camera_head.py - World derivation
hmr_backends/omega/world.py - Camera cache
- World results
Results and visualization
- Mesh overlays
hmr_backends/utils/renderer.py - World visualization
hmr_backends/omega/visualization.py - Rendered outputs
Other components
- Image or video
Connections 29
- Image or video → Pipeline CLI: provides input
- Pipeline CLI → Hand detection: runs detection
- Hand detection → YOLO detector: detects hands
- Hand detection → ViTPose merge: optionally merges
- Hand detection → Detection cache: caches detections
- Pipeline CLI → Track cleanup: runs cleanup
- Track cleanup → Detection cache: reads detections
- Track cleanup → Cleaned tracks: caches tracks
- Pipeline CLI → Mesh orchestration: starts recovery
- Mesh orchestration → Cleaned tracks: reads tracks
- Mesh orchestration → Runner factory: selects runner
- Runner factory → Batch inference: dispatches
- Runner factory → HaWoR inference: dispatches
- Batch inference → Sequence crops: prepares batches
- Batch inference → HMR backends: runs inference
- HaWoR inference → HMR backends: runs HaWoR
- HMR backends → MANO head: predicts MANO
- Mesh orchestration → Frame mesh results: writes meshes
- Pipeline CLI → Omega runtime: starts Omega pass
- Omega runtime → VGGT-Omega model: runs prediction
- Omega runtime → Camera recovery: recovers cameras
- Omega runtime → Camera cache: reuses or writes
- Omega runtime → World derivation: derives world results
- World derivation → Frame mesh results: uses mesh results
- Omega runtime → World results: writes results
- Mesh orchestration → Mesh overlays: renders overlays
- Mesh overlays → Rendered outputs: produces renders
- Omega runtime → World visualization: renders world view
- World visualization → Rendered outputs: produces visualization