# marionlepert/phantom architecture

> Architecture diagram of the GitHub repository marionlepert/phantom, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marionlepert/phantom
- Repository: https://github.com/marionlepert/phantom

## Overview

Phantom and Masquerade turn human demonstration videos into robotized training demonstrations. The user selects a dataset and processing modes through `phantom/process_data.py`; documented stages detect hands, estimate poses, segment arms, derive and smooth robot actions, remove the human arm, and render a virtual robot overlay. The map also shows shared processing infrastructure and robot twins. The README documents the workflow and capabilities; some stage internals were not sampled, so their connections are shown only where the README supports them.

Main entry points: [`phantom/process_data.py`](https://github.com/marionlepert/phantom/blob/main/phantom/process_data.py) (Pipeline runner), [`phantom/processors/action_processor.py`](https://github.com/marionlepert/phantom/blob/main/phantom/processors/action_processor.py) (Action retargeting), [`phantom/processors/robotinpaint_processor.py`](https://github.com/marionlepert/phantom/blob/main/phantom/processors/robotinpaint_processor.py) (Robot video overlay), [`phantom/processors/hand_processor.py`](https://github.com/marionlepert/phantom/blob/main/phantom/processors/hand_processor.py) (Hand pose)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Video processing"]
  node_runner["Pipeline runner<br/>[process_data.py]"]
end

subgraph group_perception["Hand perception"]
  node_bbox["Hand detection<br/>[bbox_processor.py]"]
  node_detectors["Detection models<br/>[detector_dino.py]"]
  node_handpose["Hand pose<br/>[hand_processor.py]"]
  node_hamer["HaMeR estimator<br/>[detector_hamer.py]"]
  node_handmodel["Hand kinematics<br/>[hand.py]"]
  node_segmentation["Arm segmentation"]
  node_sam2["SAM2 segmenter<br/>[detector_sam2.py]"]
end

subgraph group_robotization["Robotization"]
  node_action["Action retargeting"]
  node_smoothing["Pose smoothing"]
  node_handinpaint["Human-arm inpainting"]
  node_robotoverlay["Robot video overlay"]
  node_twin["Robot twins<br/>[twin_robot.py]"]
  node_bimanual["Bimanual twin"]
end

subgraph group_shared["Shared infrastructure"]
  node_sequences["Sequence data<br/>[phantom_data.py]"]
  node_base["Processor base<br/>[base_processor.py]"]
  node_paths["Data paths<br/>[paths.py]"]
end

node_user(("Dataset user"))
node_training[("Training outputs")]

node_user -->|"selects modes"| node_runner
node_runner -->|"runs stages"| node_bbox
node_runner -->|"runs stages"| node_handpose
node_runner -->|"runs stages"| node_segmentation
node_runner -->|"runs stages"| node_action
node_runner -->|"runs stages"| node_smoothing
node_runner -->|"runs stages"| node_handinpaint
node_runner -->|"runs stages"| node_robotoverlay
node_bbox -->|"uses detector"| node_detectors
node_handpose -->|"estimates pose"| node_hamer
node_handpose -.->|"models hand"| node_handmodel
node_segmentation -->|"segments masks"| node_sam2
node_action -.->|"retargets poses"| node_handmodel
node_action -->|"stores actions"| node_sequences
node_smoothing -.->|"stores trajectories"| node_sequences
node_robotoverlay -->|"renders robot"| node_twin
node_robotoverlay -->|"renders bimanual"| node_bimanual
node_robotoverlay -->|"writes overlays"| node_training
node_action -->|"writes actions"| node_training
node_runner -->|"configures processors"| node_base
node_base -->|"creates paths"| node_paths

click node_runner "https://github.com/marionlepert/phantom/blob/main/phantom/process_data.py"
click node_bbox "https://github.com/marionlepert/phantom/blob/main/phantom/processors/bbox_processor.py"
click node_detectors "https://github.com/marionlepert/phantom/blob/main/phantom/detectors/detector_dino.py"
click node_handpose "https://github.com/marionlepert/phantom/blob/main/phantom/processors/hand_processor.py"
click node_hamer "https://github.com/marionlepert/phantom/blob/main/phantom/detectors/detector_hamer.py"
click node_handmodel "https://github.com/marionlepert/phantom/blob/main/phantom/hand.py"
click node_segmentation "https://github.com/marionlepert/phantom/blob/main/phantom/processors/segmentation_processor.py"
click node_sam2 "https://github.com/marionlepert/phantom/blob/main/phantom/detectors/detector_sam2.py"
click node_action "https://github.com/marionlepert/phantom/blob/main/phantom/processors/action_processor.py"
click node_smoothing "https://github.com/marionlepert/phantom/blob/main/phantom/processors/smoothing_processor.py"
click node_handinpaint "https://github.com/marionlepert/phantom/blob/main/phantom/processors/handinpaint_processor.py"
click node_robotoverlay "https://github.com/marionlepert/phantom/blob/main/phantom/processors/robotinpaint_processor.py"
click node_twin "https://github.com/marionlepert/phantom/blob/main/phantom/twin_robot.py"
click node_bimanual "https://github.com/marionlepert/phantom/blob/main/phantom/twin_bimanual_robot.py"
click node_sequences "https://github.com/marionlepert/phantom/blob/main/phantom/processors/phantom_data.py"
click node_base "https://github.com/marionlepert/phantom/blob/main/phantom/processors/base_processor.py"
click node_paths "https://github.com/marionlepert/phantom/blob/main/phantom/processors/paths.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_user toneBlue
class node_bbox,node_detectors,node_handpose,node_hamer,node_handmodel,node_segmentation,node_sam2,node_training toneAmber
class node_action,node_smoothing,node_handinpaint,node_robotoverlay,node_twin,node_bimanual toneMint
class node_sequences,node_base,node_paths toneRose
```

## Components

### Video processing

- [Pipeline runner](https://github.com/marionlepert/phantom/blob/main/phantom/process_data.py): `phantom/process_data.py`

### Hand perception

- [Hand detection](https://github.com/marionlepert/phantom/blob/main/phantom/processors/bbox_processor.py): `phantom/processors/bbox_processor.py`
- [Detection models](https://github.com/marionlepert/phantom/blob/main/phantom/detectors/detector_dino.py): `phantom/detectors/detector_dino.py`
- [Hand pose](https://github.com/marionlepert/phantom/blob/main/phantom/processors/hand_processor.py): `phantom/processors/hand_processor.py`
- [HaMeR estimator](https://github.com/marionlepert/phantom/blob/main/phantom/detectors/detector_hamer.py): `phantom/detectors/detector_hamer.py`
- [Hand kinematics](https://github.com/marionlepert/phantom/blob/main/phantom/hand.py): `phantom/hand.py`
- [Arm segmentation](https://github.com/marionlepert/phantom/blob/main/phantom/processors/segmentation_processor.py): `phantom/processors/segmentation_processor.py`
- [SAM2 segmenter](https://github.com/marionlepert/phantom/blob/main/phantom/detectors/detector_sam2.py): `phantom/detectors/detector_sam2.py`

### Robotization

- [Action retargeting](https://github.com/marionlepert/phantom/blob/main/phantom/processors/action_processor.py): `phantom/processors/action_processor.py`
- [Pose smoothing](https://github.com/marionlepert/phantom/blob/main/phantom/processors/smoothing_processor.py): `phantom/processors/smoothing_processor.py`
- [Human-arm inpainting](https://github.com/marionlepert/phantom/blob/main/phantom/processors/handinpaint_processor.py): `phantom/processors/handinpaint_processor.py`
- [Robot video overlay](https://github.com/marionlepert/phantom/blob/main/phantom/processors/robotinpaint_processor.py): `phantom/processors/robotinpaint_processor.py`
- [Robot twins](https://github.com/marionlepert/phantom/blob/main/phantom/twin_robot.py): `phantom/twin_robot.py`
- [Bimanual twin](https://github.com/marionlepert/phantom/blob/main/phantom/twin_bimanual_robot.py): `phantom/twin_bimanual_robot.py`

### Shared infrastructure

- [Sequence data](https://github.com/marionlepert/phantom/blob/main/phantom/processors/phantom_data.py): `phantom/processors/phantom_data.py`
- [Processor base](https://github.com/marionlepert/phantom/blob/main/phantom/processors/base_processor.py): `phantom/processors/base_processor.py`
- [Data paths](https://github.com/marionlepert/phantom/blob/main/phantom/processors/paths.py): `phantom/processors/paths.py`

### Other components

- **Dataset user**
- **Training outputs**

## Connections

- Dataset user → Pipeline runner: selects modes (evidence: `README.md`)
- Pipeline runner → Hand detection: runs stages (evidence: `README.md`)
- Pipeline runner → Hand pose: runs stages (evidence: `README.md`)
- Pipeline runner → Arm segmentation: runs stages (evidence: `README.md`)
- Pipeline runner → Action retargeting: runs stages (evidence: `README.md`)
- Pipeline runner → Pose smoothing: runs stages (evidence: `README.md`)
- Pipeline runner → Human-arm inpainting: runs stages (evidence: `README.md`)
- Pipeline runner → Robot video overlay: runs stages (evidence: `README.md`)
- Hand detection → Detection models: uses detector (evidence: `phantom/processors/bbox_processor.py`)
- Hand pose → HaMeR estimator: estimates pose (evidence: `phantom/processors/hand_processor.py`)
- Hand pose → Hand kinematics: models hand
- Arm segmentation → SAM2 segmenter: segments masks (evidence: `phantom/processors/segmentation_processor.py`)
- Action retargeting → Hand kinematics: retargets poses (evidence: `phantom/processors/action_processor.py`)
- Action retargeting → Sequence data: stores actions (evidence: `phantom/processors/action_processor.py`)
- Pose smoothing → Sequence data: stores trajectories
- Robot video overlay → Robot twins: renders robot (evidence: `phantom/processors/robotinpaint_processor.py`)
- Robot video overlay → Bimanual twin: renders bimanual (evidence: `phantom/processors/robotinpaint_processor.py`)
- Robot video overlay → Training outputs: writes overlays (evidence: `README.md`)
- Action retargeting → Training outputs: writes actions (evidence: `README.md`)
- Pipeline runner → Processor base: configures processors (evidence: `phantom/process_data.py`)
- Processor base → Data paths: creates paths (evidence: `phantom/processors/base_processor.py`)