How z1hanw/crisp-real2sim works
CRISP turns monocular video into human motion and planar scene assets, then transfers them into a motion-tracking simulation workflow. The README documents the reconstruction stages and MotionTracking training/evaluation; the sampled source confirms the NeuralPlane training components and Viser viewer wiring. Some reconstruction stages are documented capabilities without sampled implementation excerpts, so their links are shown only where the README describes the sequence. NKSR and contact hallucination are optional branches.
Main entry points: App.tsx Viewer App · neuralplane_pipeline.py Scene Reconstruction · neuralplane_datamanager.py Planar Data · train_agent.py Train and Evaluate
Components 25
Video Reconstruction
- Video Preparation
MotionTracking/scripts/video2npy.py - Human Reconstruction
vis_scripts/viser_m/neuralplane/frontend/monocular - Scene Reconstruction
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py - Contact Hallucination
vis_scripts/viser_m - NKSR Surface
vis_scripts/viser_m/run_nksr.py
Planar Scene Learning
- Planar Data
vis_scripts/viser_m/neuralplane/neuralplane_datamanager.py - NeuralPlane Model
vis_scripts/viser_m/neuralplane/neuralplane_model.py - Training Pipeline
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py
Motion Tracking
- CRISP-to-RL Bridge
MotionTracking/scripts/bridge_crisp_to_motiontracking.py - Motion Library
MotionTracking/motion_tracking/utils/motion_lib.py - Skeleton Motion
MotionTracking/poselib/poselib/skeleton/skeleton3d.py - Humanoid Environment
MotionTracking/motion_tracking/envs/isaacgym/mimic_humanoid.py - Mimic Agent
MotionTracking/motion_tracking/agents/mimic.py - PPO Trainer
MotionTracking/motion_tracking/agents/ppo.py - Train and Evaluate
MotionTracking/motion_tracking/train_agent.py
Visualization
- Viewer App
vis_scripts/viser_m/src/viser/client/src/App.tsx - Scene Tree
vis_scripts/viser/src/viser/client/src/SceneTree.tsx - Scene State
vis_scripts/viser/src/viser/client/src/SceneTreeState.tsx - GUI State
vis_scripts/viser/src/viser/client/src/ControlPanel/GuiState.tsx - WebSocket Messages
vis_scripts/viser/src/viser/client/src/WebsocketMessages.ts - Splat Sort Worker
vis_scripts/viser/src/viser/client/src/Splatting/SplatSortWorker.ts - WASM Sorter
vis_scripts/viser/src/viser/client/src/Splatting/WasmSorter/Sorter.mjs
Other components
- Video Sequence
- Viewer User
- Browser Client
Connections 23
- Video Sequence → Video Preparation: provides video
- Video Preparation → Human Reconstruction: prepares frames
- Human Reconstruction → Scene Reconstruction: aligns reconstruction
- Scene Reconstruction → Planar Data: provides scene data
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py - Training Pipeline → Planar Data: loads batches
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py - Training Pipeline → NeuralPlane Model: computes losses
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py - Planar Data → Training Pipeline: supplies rays
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py - Training Pipeline → Planar Data: updates primitives
vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py - Video Sequence → Contact Hallucination: optionally adds
- Scene Reconstruction → NKSR Surface: optionally refines
- Scene Reconstruction → CRISP-to-RL Bridge: post-processes assets
- CRISP-to-RL Bridge → Motion Library: transfers motion
- Motion Library → Skeleton Motion: uses skeleton data
MotionTracking/motion_tracking/utils/mimic_motion_authoring_motion_lib.py - Train and Evaluate → Humanoid Environment: runs simulation
- Train and Evaluate → Mimic Agent: trains agent
- Train and Evaluate → PPO Trainer: optimizes policy
- Viewer User → Viewer App: opens viewer
- Viewer App → Scene Tree: renders scene
vis_scripts/viser_m/src/viser/client/src/App.tsx - Viewer App → WebSocket Messages: receives updates
vis_scripts/viser_m/src/viser/client/src/App.tsx - Viewer App → Browser Client: renders interface
- Scene Tree → Scene State: reads state
- Viewer App → GUI State: uses GUI state
vis_scripts/viser_m/src/viser/client/src/App.tsx - Splat Sort Worker → WASM Sorter: invokes sorter
vis_scripts/viser/src/viser/client/src/Splatting/SplatSortWorker.ts