How marionlepert/phantom works
Phantom and Masquerade turn human demonstration videos into robotized training demonstrations. The user selects a dataset and processing modes through phantom/process_data.py; documented stages detect hands, estimate poses, segment arms, derive and smooth robot actions, remove the human arm, and render a virtual robot overlay. The map also shows shared processing infrastructure and robot twins. The README documents the workflow and capabilities; some stage internals were not sampled, so their connections are shown only where the README supports them.
Main entry points: process_data.py Pipeline runner · action_processor.py Action retargeting · robotinpaint_processor.py Robot video overlay · hand_processor.py Hand pose
Components 19
Video processing
- Pipeline runner
phantom/process_data.py
Hand perception
- Hand detection
phantom/processors/bbox_processor.py - Detection models
phantom/detectors/detector_dino.py - Hand pose
phantom/processors/hand_processor.py - HaMeR estimator
phantom/detectors/detector_hamer.py - Hand kinematics
phantom/hand.py - Arm segmentation
phantom/processors/segmentation_processor.py - SAM2 segmenter
phantom/detectors/detector_sam2.py
Robotization
- Action retargeting
phantom/processors/action_processor.py - Pose smoothing
phantom/processors/smoothing_processor.py - Human-arm inpainting
phantom/processors/handinpaint_processor.py - Robot video overlay
phantom/processors/robotinpaint_processor.py - Robot twins
phantom/twin_robot.py - Bimanual twin
phantom/twin_bimanual_robot.py
Shared infrastructure
- Sequence data
phantom/processors/phantom_data.py - Processor base
phantom/processors/base_processor.py - Data paths
phantom/processors/paths.py
Other components
- Dataset user
- Training outputs
Connections 21
- Dataset user → Pipeline runner: selects modes
README.md - Pipeline runner → Hand detection: runs stages
README.md - Pipeline runner → Hand pose: runs stages
README.md - Pipeline runner → Arm segmentation: runs stages
README.md - Pipeline runner → Action retargeting: runs stages
README.md - Pipeline runner → Pose smoothing: runs stages
README.md - Pipeline runner → Human-arm inpainting: runs stages
README.md - Pipeline runner → Robot video overlay: runs stages
README.md - Hand detection → Detection models: uses detector
phantom/processors/bbox_processor.py - Hand pose → HaMeR estimator: estimates pose
phantom/processors/hand_processor.py - Hand pose → Hand kinematics: models hand
- Arm segmentation → SAM2 segmenter: segments masks
phantom/processors/segmentation_processor.py - Action retargeting → Hand kinematics: retargets poses
phantom/processors/action_processor.py - Action retargeting → Sequence data: stores actions
phantom/processors/action_processor.py - Pose smoothing → Sequence data: stores trajectories
- Robot video overlay → Robot twins: renders robot
phantom/processors/robotinpaint_processor.py - Robot video overlay → Bimanual twin: renders bimanual
phantom/processors/robotinpaint_processor.py - Robot video overlay → Training outputs: writes overlays
README.md - Action retargeting → Training outputs: writes actions
README.md - Pipeline runner → Processor base: configures processors
phantom/process_data.py - Processor base → Data paths: creates paths
phantom/processors/base_processor.py