# zijian-wu/egohandkit architecture

> Architecture diagram of the GitHub repository zijian-wu/egohandkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zijian-wu/egohandkit
- Repository: https://github.com/zijian-wu/egohandkit

## Overview

EgoHandKit turns an image folder or video into per-frame hand detections, cleaned tracks, backend-specific MANO mesh estimates, and—by default—Omega camera and world-space hand results with visualizations. `run.py` coordinates the four documented passes; `mesh_recovery.py` and the backend runners handle mesh inference, while the Omega runtime derives world results. Optional ViTPose-assisted detection and configurable HMR backends are shown as branches. The README documents these stages and cache behavior; sampled source confirms sequence batching and Omega visualization, but not every orchestration call, so uncertain internal wiring is omitted.

Main entry points: [`hmr_backends/omega/runner.py`](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/runner.py) (Omega runtime), [`run.py`](https://github.com/zijian-wu/egohandkit/blob/master/run.py) (Pipeline CLI), [`mesh_recovery.py`](https://github.com/zijian-wu/egohandkit/blob/master/mesh_recovery.py) (Mesh orchestration), [`bbox_utils.py`](https://github.com/zijian-wu/egohandkit/blob/master/bbox_utils.py) (Hand detection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Input and detection"]
  node_pipeline["Pipeline CLI<br/>[run.py]"]
  node_detect["Hand detection<br/>[bbox_utils.py]"]
  node_yolo["YOLO detector"]
  node_vitpose["ViTPose merge<br/>[vitpose_model.py]"]
  node_clean["Track cleanup<br/>[bbox_utils.py]"]
  node_raw_cache["Detection cache"]
  node_clean_cache["Cleaned tracks"]
end

subgraph group_mesh["Mesh recovery"]
  node_recovery["Mesh orchestration<br/>[mesh_recovery.py]"]
  node_factory["Runner factory<br/>[factory.py]"]
  node_batch_runner["Batch inference<br/>[batch_runner.py]"]
  node_hawor_runner["HaWoR inference<br/>[hawor_runner.py]"]
  node_crop_dataset["Sequence crops<br/>[vitdet_dataset.py]"]
  node_hmr_models["HMR backends<br/>[hamer.py]"]
  node_mano["MANO head<br/>[mano_head.py]"]
  node_mesh_cache["Frame mesh results"]
end

subgraph group_omega["Omega world derivation"]
  node_omega_runner["Omega runtime<br/>[runner.py]"]
  node_omega_model["VGGT-Omega model<br/>[vggt_omega.py]"]
  node_camera["Camera recovery<br/>[camera_head.py]"]
  node_world["World derivation<br/>[world.py]"]
  node_camera_cache["Camera cache"]
  node_world_cache["World results"]
end

subgraph group_results["Results and visualization"]
  node_overlay["Mesh overlays<br/>[renderer.py]"]
  node_world_vis["World visualization<br/>[visualization.py]"]
  node_outputs["Rendered outputs"]
end

node_user(("Image or video"))

node_user -->|"provides input"| node_pipeline
node_pipeline -->|"runs detection"| node_detect
node_detect -->|"detects hands"| node_yolo
node_detect -.->|"optionally merges"| node_vitpose
node_detect -->|"caches detections"| node_raw_cache
node_pipeline -->|"runs cleanup"| node_clean
node_clean -->|"reads detections"| node_raw_cache
node_clean -->|"caches tracks"| node_clean_cache
node_pipeline -->|"starts recovery"| node_recovery
node_recovery -->|"reads tracks"| node_clean_cache
node_recovery -->|"selects runner"| node_factory
node_factory -->|"dispatches"| node_batch_runner
node_factory -->|"dispatches"| node_hawor_runner
node_batch_runner -->|"prepares batches"| node_crop_dataset
node_batch_runner -->|"runs inference"| node_hmr_models
node_hawor_runner -->|"runs HaWoR"| node_hmr_models
node_hmr_models -->|"predicts MANO"| node_mano
node_recovery -->|"writes meshes"| node_mesh_cache
node_pipeline -->|"starts Omega pass"| node_omega_runner
node_omega_runner -->|"runs prediction"| node_omega_model
node_omega_runner -->|"recovers cameras"| node_camera
node_omega_runner -->|"reuses or writes"| node_camera_cache
node_omega_runner -->|"derives world results"| node_world
node_world -->|"uses mesh results"| node_mesh_cache
node_omega_runner -->|"writes results"| node_world_cache
node_recovery -->|"renders overlays"| node_overlay
node_overlay -->|"produces renders"| node_outputs
node_omega_runner -->|"renders world view"| node_world_vis
node_world_vis -->|"produces visualization"| node_outputs

click node_pipeline "https://github.com/zijian-wu/egohandkit/blob/master/run.py"
click node_detect "https://github.com/zijian-wu/egohandkit/blob/master/bbox_utils.py"
click node_vitpose "https://github.com/zijian-wu/egohandkit/blob/master/vitpose_model.py"
click node_clean "https://github.com/zijian-wu/egohandkit/blob/master/bbox_utils.py"
click node_recovery "https://github.com/zijian-wu/egohandkit/blob/master/mesh_recovery.py"
click node_factory "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/runners/factory.py"
click node_batch_runner "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/runners/batch_runner.py"
click node_hawor_runner "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/runners/hawor_runner.py"
click node_crop_dataset "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/datasets/vitdet_dataset.py"
click node_hmr_models "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/models/hamer.py"
click node_mano "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/models/heads/mano_head.py"
click node_omega_runner "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/runner.py"
click node_omega_model "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/models/vggt_omega.py"
click node_camera "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/models/heads/camera_head.py"
click node_world "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/world.py"
click node_overlay "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/utils/renderer.py"
click node_world_vis "https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/visualization.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_detect,node_yolo,node_vitpose,node_clean,node_raw_cache,node_clean_cache toneBlue
class node_recovery,node_factory,node_batch_runner,node_hawor_runner,node_crop_dataset,node_hmr_models,node_mano,node_mesh_cache toneAmber
class node_omega_runner,node_omega_model,node_camera,node_world,node_camera_cache,node_world_cache toneMint
class node_overlay,node_world_vis,node_outputs toneRose
class node_user toneIndigo
```

## Components

### Input and detection

- [Pipeline CLI](https://github.com/zijian-wu/egohandkit/blob/master/run.py): `run.py`
- [Hand detection](https://github.com/zijian-wu/egohandkit/blob/master/bbox_utils.py): `bbox_utils.py`
- **YOLO detector**
- [ViTPose merge](https://github.com/zijian-wu/egohandkit/blob/master/vitpose_model.py): `vitpose_model.py`
- [Track cleanup](https://github.com/zijian-wu/egohandkit/blob/master/bbox_utils.py): `bbox_utils.py`
- **Detection cache**
- **Cleaned tracks**

### Mesh recovery

- [Mesh orchestration](https://github.com/zijian-wu/egohandkit/blob/master/mesh_recovery.py): `mesh_recovery.py`
- [Runner factory](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/runners/factory.py): `hmr_backends/runners/factory.py`
- [Batch inference](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/runners/batch_runner.py): `hmr_backends/runners/batch_runner.py`
- [HaWoR inference](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/runners/hawor_runner.py): `hmr_backends/runners/hawor_runner.py`
- [Sequence crops](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/datasets/vitdet_dataset.py): `hmr_backends/datasets/vitdet_dataset.py`
- [HMR backends](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/models/hamer.py): `hmr_backends/models/hamer.py`
- [MANO head](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/models/heads/mano_head.py): `hmr_backends/models/heads/mano_head.py`
- **Frame mesh results**

### Omega world derivation

- [Omega runtime](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/runner.py): `hmr_backends/omega/runner.py`
- [VGGT-Omega model](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/models/vggt_omega.py): `hmr_backends/omega/models/vggt_omega.py`
- [Camera recovery](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/models/heads/camera_head.py): `hmr_backends/omega/models/heads/camera_head.py`
- [World derivation](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/world.py): `hmr_backends/omega/world.py`
- **Camera cache**
- **World results**

### Results and visualization

- [Mesh overlays](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/utils/renderer.py): `hmr_backends/utils/renderer.py`
- [World visualization](https://github.com/zijian-wu/egohandkit/blob/master/hmr_backends/omega/visualization.py): `hmr_backends/omega/visualization.py`
- **Rendered outputs**

### Other components

- **Image or video**

## Connections

- Image or video → Pipeline CLI: provides input
- Pipeline CLI → Hand detection: runs detection
- Hand detection → YOLO detector: detects hands
- Hand detection → ViTPose merge: optionally merges
- Hand detection → Detection cache: caches detections
- Pipeline CLI → Track cleanup: runs cleanup
- Track cleanup → Detection cache: reads detections
- Track cleanup → Cleaned tracks: caches tracks
- Pipeline CLI → Mesh orchestration: starts recovery
- Mesh orchestration → Cleaned tracks: reads tracks
- Mesh orchestration → Runner factory: selects runner
- Runner factory → Batch inference: dispatches
- Runner factory → HaWoR inference: dispatches
- Batch inference → Sequence crops: prepares batches
- Batch inference → HMR backends: runs inference
- HaWoR inference → HMR backends: runs HaWoR
- HMR backends → MANO head: predicts MANO
- Mesh orchestration → Frame mesh results: writes meshes
- Pipeline CLI → Omega runtime: starts Omega pass
- Omega runtime → VGGT-Omega model: runs prediction
- Omega runtime → Camera recovery: recovers cameras
- Omega runtime → Camera cache: reuses or writes
- Omega runtime → World derivation: derives world results
- World derivation → Frame mesh results: uses mesh results
- Omega runtime → World results: writes results
- Mesh orchestration → Mesh overlays: renders overlays
- Mesh overlays → Rendered outputs: produces renders
- Omega runtime → World visualization: renders world view
- World visualization → Rendered outputs: produces visualization