# thundervvv/hawor architecture

> Architecture diagram of the GitHub repository thundervvv/hawor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thundervvv/hawor
- Repository: https://github.com/thundervvv/hawor

## Overview

##### Project Type and Purpose
- This is a computer vision and deep learning research project aimed at reconstructing hand motion in world coordinates from video.
- It integrates several components: a hand reconstruction model (HaWoR), supplementary modules for in-between frame infilling (the “infiller” module), camera motion estimation (via integration with thirdparty/DROID-SLAM), and additional depth/monocular cues from Metric3D.
- The README emphasizes both demonstration (via demo.py and app.py) and training/inference pipelines.

##### File Structure and Architectural Clues
- Top-level Files and Folders:
– Main entry points: app.py and demo.py (for visualization/demo purposes).
– Directories like hawor, infiller, and lib contain core modules. For example:
▪ hawor/ hosts the main configurations and utility functions related to the HaWoR pipeline.
▪ infiller/ holds code related to “inbetweening” hand motion and supplementary processing like geometry and model modules.
▪ lib/ encompasses diverse core utilities, datasets, models (with backbones and transformer components), pipelines, and visualization tools.
– The _DATA folder houses hand model (MANO) files required to drive the reconstruction.
– The thirdparty folder brings in external dependencies such as DROID-SLAM and Metric3D—components critical for camera pose estimation and depth processing.
– The scripts directory signals tooling for testing (e.g., scripts_test_video).

##### Key Components and Their Interactions
a. Input / Preprocessing:
– Video Input: The demo scripts (demo.py, app.py) ingest video data (examples in the example/ folder).
– Data Processing Utilities: Modules in hawor/utils, lib/utils, and infiller/hand_utils preprocess frames (geometry, rotation, processing).

b. SLAM and Camera Motion Estimation:
– Third-party Integration: The thirdparty/DROID-SLAM component is installed and used to extract camera pose and scene structure information.
– These SLAM outputs feed into the overall reconstruction pipeline.

c. Hand Reconstruction Pipeline:
- HaWoR Core Module: Housed in the hawor and lib/models directories, this is the central hand reconstruction engine operating on processed input.
- Infiller Module: Located in infiller/, it likely refines or interpolates hand pose data between frames.
- MANO Integration: Data in _DATA ensures that the hand model (MANO_RIGHT.pkl and MANO_LEFT.pkl) is used as a priors/input for reconstruction.

d. Visualization and Output:
– Rendering Components: Found in hawor/utils/render_openpose.py and lib/vis (renderer, viewer, etc.), these modules handle the rendering of reconstructed hand motions either in camera view or world space.
– Demo and Testing: The provided demo.py and scripts give examples of how to run the complete pipeline, from video input to visualization.

##### Architectural Patterns & Important Design Principles
- Modular Design: Each functional aspect (data I/O, camera pose estimation, hand reconstruction, visualization) is encapsulated in its own module or subpackage.
- Separation of Concerns: Core processing (hawor, infiller, lib) is distinct from third-party software (DROID-SLAM, Metric3D). This separation allows independent development and testing.
- Pipeline Pattern: Data flows sequentially from input, through preprocessing, through modular computer vision and deep learning subcomponents, to final visualization.
- External Dependencies: Explicit integration steps are needed for installing and configuring third-party modules and models (MANO, DROID-SLAM weights, Metric3D weights).

##### Elements
- Clearly labeled components:
▪ Video Input / Preprocessing Module
▪ SLAM Module (DROID-SLAM)
▪ Hand Reconstruction Engine (HaWoR’s core modules in hawor and lib/models)
▪ In-between Motion Refinement (infiller)
▪ Visualization/Rendering (lib/vis, hawor/utils/render_openpose)
▪ External Data/Weights (MANO models in _DATA, weights in weights directories)
– From video input to preprocessing, then branching out to both SLAM (for camera pose) and direct hand image feature extraction.
– Merging of SLAM outputs with hand reconstruction pipelines.
– Output flow towards visualization and demo interfaces.

##### How to Proceed
– Add submodules within the Processing Pipeline:
- One branch for SLAM (camera motion estimation),
- One branch for Hand Reconstruction (HaWoR and Infiller) which integrates cues from the SLAM module.
– Clearly annotate external integrations (e.g., installation setup for DROID-SLAM and Metric3D) and data repositories (MANO models).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        VideoInput["Video Input / Preprocessing"]:::input
    end

    %% Processing Pipeline
    subgraph "Processing Pipeline"
        DataProc["Data Processing Utilities"]:::processing
        SLAM["SLAM Module (DROID-SLAM)"]:::external
        SLAMScript["SLAM Testing Script"]:::external
        HandRec["Hand Reconstruction Engine (HaWoR Core)"]:::processing
        Infiller["In-Between Motion Refinement (Infiller)"]:::processing
        MANO["MANO Integration (Hand Model Data)"]:::external
        Metric3D["Depth/Monocular Cues (Metric3D)"]:::external
    end

    %% Visualization Layer
    subgraph "Visualization Layer"
        VisRender["Visualization & Rendering"]:::visualization
    end

    %% Data Flow Connections
    VideoInput -->|"preprocesses"| DataProc
    DataProc -->|"features"| SLAM
    DataProc -->|"features"| HandRec
    SLAM -->|"poseInfo"| HandRec
    SLAM -->|"tests"| SLAMScript
    HandRec -->|"refines"| Infiller
    MANO -->|"providesPriors"| HandRec
    Metric3D -->|"enhances"| HandRec
    HandRec -->|"outputs"| VisRender
    Infiller -->|"refinedOutput"| VisRender

    %% Click Events for Component Mapping
    click VideoInput "https://github.com/thundervvv/hawor/blob/main/app.py"
    click VideoInput "https://github.com/thundervvv/hawor/blob/main/demo.py"
    click VideoInput "https://github.com/thundervvv/hawor/tree/main/example/"
    
    click DataProc "https://github.com/thundervvv/hawor/tree/main/hawor/utils"
    click DataProc "https://github.com/thundervvv/hawor/tree/main/lib/utils"
    click DataProc "https://github.com/thundervvv/hawor/tree/main/infiller/hand_utils"
    
    click SLAM "https://github.com/thundervvv/hawor/tree/main/thirdparty/DROID-SLAM"
    click SLAMScript "https://github.com/thundervvv/hawor/blob/main/scripts/scripts_test_video/hawor_slam.py"
    
    click HandRec "https://github.com/thundervvv/hawor/tree/main/hawor/"
    click HandRec "https://github.com/thundervvv/hawor/blob/main/lib/models/hawor.py"
    
    click Infiller "https://github.com/thundervvv/hawor/tree/main/infiller/hand_utils"
    click Infiller "https://github.com/thundervvv/hawor/tree/main/infiller/lib/model"
    
    click MANO "https://github.com/thundervvv/hawor/tree/main/_DATA/data/mano"
    click MANO "https://github.com/thundervvv/hawor/tree/main/_DATA/data_left/mano_left"
    
    click VisRender "https://github.com/thundervvv/hawor/blob/main/hawor/utils/render_openpose.py"
    click VisRender "https://github.com/thundervvv/hawor/tree/main/lib/vis"
    
    click Metric3D "https://github.com/thundervvv/hawor/tree/main/thirdparty/Metric3D"

    %% Class Definitions for Color Styling
    classDef input fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef processing fill:#ccffcc,stroke:#008000,stroke-width:2px;
    classDef external fill:#ccccff,stroke:#0000cc,stroke-width:2px;
    classDef visualization fill:#ffffcc,stroke:#cccc00,stroke-width:2px;
```
