# z1hanw/crisp-real2sim architecture

> Architecture diagram of the GitHub repository z1hanw/crisp-real2sim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/z1hanw/crisp-real2sim
- Repository: https://github.com/z1hanw/crisp-real2sim

## Overview

CRISP turns monocular video into human motion and planar scene assets, then transfers them into a motion-tracking simulation workflow. The README documents the reconstruction stages and MotionTracking training/evaluation; the sampled source confirms the NeuralPlane training components and Viser viewer wiring. Some reconstruction stages are documented capabilities without sampled implementation excerpts, so their links are shown only where the README describes the sequence. NKSR and contact hallucination are optional branches.

Main entry points: [`vis_scripts/viser_m/src/viser/client/src/App.tsx`](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/src/viser/client/src/App.tsx) (Viewer App), [`vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py) (Scene Reconstruction), [`vis_scripts/viser_m/neuralplane/neuralplane_datamanager.py`](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_datamanager.py) (Planar Data), [`MotionTracking/motion_tracking/train_agent.py`](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/train_agent.py) (Train and Evaluate)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_reconstruction["Video Reconstruction"]
  node_video_prep["Video Preparation<br/>[video2npy.py]"]
  node_human_reconstruction["Human Reconstruction"]
  node_scene_reconstruction["Scene Reconstruction"]
  node_contact["Contact Hallucination"]
  node_nksr["NKSR Surface<br/>[run_nksr.py]"]
end

subgraph group_planar["Planar Scene Learning"]
  node_planar_data["Planar Data"]
  node_planar_model["NeuralPlane Model"]
  node_planar_pipeline["Training Pipeline"]
end

subgraph group_tracking["Motion Tracking"]
  node_real2sim_bridge["CRISP-to-RL Bridge"]
  node_motion_data["Motion Library<br/>[motion_lib.py]"]
  node_skeleton["Skeleton Motion<br/>[skeleton3d.py]"]
  node_sim_environment["Humanoid Environment<br/>[mimic_humanoid.py]"]
  node_agent["Mimic Agent<br/>[mimic.py]"]
  node_ppo["PPO Trainer<br/>[ppo.py]"]
  node_train_eval["Train and Evaluate<br/>[train_agent.py]"]
end

subgraph group_viewer["Visualization"]
  node_viser_app["Viewer App<br/>[App.tsx]"]
  node_scene_tree["Scene Tree<br/>[SceneTree.tsx]"]
  node_scene_state["Scene State<br/>[SceneTreeState.tsx]"]
  node_gui_state["GUI State<br/>[GuiState.tsx]"]
  node_websocket["WebSocket Messages"]
  node_splat_worker["Splat Sort Worker<br/>[SplatSortWorker.ts]"]
  node_sorter["WASM Sorter<br/>[Sorter.mjs]"]
end

node_video_user(("Video Sequence"))
node_viewer_user(("Viewer User"))
node_browser(("Browser Client"))

node_video_user -->|"provides video"| node_video_prep
node_video_prep -->|"prepares frames"| node_human_reconstruction
node_human_reconstruction -->|"aligns reconstruction"| node_scene_reconstruction
node_scene_reconstruction -->|"provides scene data"| node_planar_data
node_planar_pipeline -->|"loads batches"| node_planar_data
node_planar_pipeline -->|"computes losses"| node_planar_model
node_planar_data -->|"supplies rays"| node_planar_pipeline
node_planar_pipeline -->|"updates primitives"| node_planar_data
node_video_user -.->|"optionally adds"| node_contact
node_scene_reconstruction -.->|"optionally refines"| node_nksr
node_scene_reconstruction -->|"post-processes assets"| node_real2sim_bridge
node_real2sim_bridge -->|"transfers motion"| node_motion_data
node_motion_data -->|"uses skeleton data"| node_skeleton
node_train_eval -.->|"runs simulation"| node_sim_environment
node_train_eval -.->|"trains agent"| node_agent
node_train_eval -.->|"optimizes policy"| node_ppo
node_viewer_user -->|"opens viewer"| node_viser_app
node_viser_app -->|"renders scene"| node_scene_tree
node_viser_app -->|"receives updates"| node_websocket
node_viser_app -.->|"renders interface"| node_browser
node_scene_tree -.->|"reads state"| node_scene_state
node_viser_app -->|"uses GUI state"| node_gui_state
node_splat_worker -->|"invokes sorter"| node_sorter

click node_video_prep "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/scripts/video2npy.py"
click node_human_reconstruction "https://github.com/z1hanw/crisp-real2sim/tree/main/vis_scripts/viser_m/neuralplane/frontend/monocular"
click node_scene_reconstruction "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py"
click node_planar_data "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_datamanager.py"
click node_planar_model "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_model.py"
click node_planar_pipeline "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py"
click node_contact "https://github.com/z1hanw/crisp-real2sim/tree/main/vis_scripts/viser_m"
click node_nksr "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/run_nksr.py"
click node_real2sim_bridge "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/scripts/bridge_crisp_to_motiontracking.py"
click node_motion_data "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/utils/motion_lib.py"
click node_skeleton "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/poselib/poselib/skeleton/skeleton3d.py"
click node_sim_environment "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/envs/isaacgym/mimic_humanoid.py"
click node_agent "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/agents/mimic.py"
click node_ppo "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/agents/ppo.py"
click node_train_eval "https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/train_agent.py"
click node_viser_app "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/src/viser/client/src/App.tsx"
click node_scene_tree "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/SceneTree.tsx"
click node_scene_state "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/SceneTreeState.tsx"
click node_gui_state "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/ControlPanel/GuiState.tsx"
click node_websocket "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/WebsocketMessages.ts"
click node_splat_worker "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/Splatting/SplatSortWorker.ts"
click node_sorter "https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/Splatting/WasmSorter/Sorter.mjs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_video_prep,node_human_reconstruction,node_scene_reconstruction,node_contact,node_nksr,node_viewer_user,node_browser toneBlue
class node_planar_data,node_planar_model,node_planar_pipeline toneAmber
class node_real2sim_bridge,node_motion_data,node_skeleton,node_sim_environment,node_agent,node_ppo,node_train_eval toneMint
class node_viser_app,node_scene_tree,node_scene_state,node_gui_state,node_websocket,node_splat_worker,node_sorter toneRose
class node_video_user toneIndigo
```

## Components

### Video Reconstruction

- [Video Preparation](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/scripts/video2npy.py): `MotionTracking/scripts/video2npy.py`
- [Human Reconstruction](https://github.com/z1hanw/crisp-real2sim/tree/main/vis_scripts/viser_m/neuralplane/frontend/monocular): `vis_scripts/viser_m/neuralplane/frontend/monocular`
- [Scene Reconstruction](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py): `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`
- [Contact Hallucination](https://github.com/z1hanw/crisp-real2sim/tree/main/vis_scripts/viser_m): `vis_scripts/viser_m`
- [NKSR Surface](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/run_nksr.py): `vis_scripts/viser_m/run_nksr.py`

### Planar Scene Learning

- [Planar Data](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_datamanager.py): `vis_scripts/viser_m/neuralplane/neuralplane_datamanager.py`
- [NeuralPlane Model](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_model.py): `vis_scripts/viser_m/neuralplane/neuralplane_model.py`
- [Training Pipeline](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py): `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`

### Motion Tracking

- [CRISP-to-RL Bridge](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/scripts/bridge_crisp_to_motiontracking.py): `MotionTracking/scripts/bridge_crisp_to_motiontracking.py`
- [Motion Library](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/utils/motion_lib.py): `MotionTracking/motion_tracking/utils/motion_lib.py`
- [Skeleton Motion](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/poselib/poselib/skeleton/skeleton3d.py): `MotionTracking/poselib/poselib/skeleton/skeleton3d.py`
- [Humanoid Environment](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/envs/isaacgym/mimic_humanoid.py): `MotionTracking/motion_tracking/envs/isaacgym/mimic_humanoid.py`
- [Mimic Agent](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/agents/mimic.py): `MotionTracking/motion_tracking/agents/mimic.py`
- [PPO Trainer](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/agents/ppo.py): `MotionTracking/motion_tracking/agents/ppo.py`
- [Train and Evaluate](https://github.com/z1hanw/crisp-real2sim/blob/main/MotionTracking/motion_tracking/train_agent.py): `MotionTracking/motion_tracking/train_agent.py`

### Visualization

- [Viewer App](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser_m/src/viser/client/src/App.tsx): `vis_scripts/viser_m/src/viser/client/src/App.tsx`
- [Scene Tree](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/SceneTree.tsx): `vis_scripts/viser/src/viser/client/src/SceneTree.tsx`
- [Scene State](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/SceneTreeState.tsx): `vis_scripts/viser/src/viser/client/src/SceneTreeState.tsx`
- [GUI State](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/ControlPanel/GuiState.tsx): `vis_scripts/viser/src/viser/client/src/ControlPanel/GuiState.tsx`
- [WebSocket Messages](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/WebsocketMessages.ts): `vis_scripts/viser/src/viser/client/src/WebsocketMessages.ts`
- [Splat Sort Worker](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/Splatting/SplatSortWorker.ts): `vis_scripts/viser/src/viser/client/src/Splatting/SplatSortWorker.ts`
- [WASM Sorter](https://github.com/z1hanw/crisp-real2sim/blob/main/vis_scripts/viser/src/viser/client/src/Splatting/WasmSorter/Sorter.mjs): `vis_scripts/viser/src/viser/client/src/Splatting/WasmSorter/Sorter.mjs`

### Other components

- **Video Sequence**
- **Viewer User**
- **Browser Client**

## Connections

- Video Sequence → Video Preparation: provides video
- Video Preparation → Human Reconstruction: prepares frames
- Human Reconstruction → Scene Reconstruction: aligns reconstruction
- Scene Reconstruction → Planar Data: provides scene data (evidence: `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`)
- Training Pipeline → Planar Data: loads batches (evidence: `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`)
- Training Pipeline → NeuralPlane Model: computes losses (evidence: `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`)
- Planar Data → Training Pipeline: supplies rays (evidence: `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`)
- Training Pipeline → Planar Data: updates primitives (evidence: `vis_scripts/viser_m/neuralplane/neuralplane_pipeline.py`)
- Video Sequence → Contact Hallucination: optionally adds
- Scene Reconstruction → NKSR Surface: optionally refines
- Scene Reconstruction → CRISP-to-RL Bridge: post-processes assets
- CRISP-to-RL Bridge → Motion Library: transfers motion
- Motion Library → Skeleton Motion: uses skeleton data (evidence: `MotionTracking/motion_tracking/utils/mimic_motion_authoring_motion_lib.py`)
- Train and Evaluate → Humanoid Environment: runs simulation
- Train and Evaluate → Mimic Agent: trains agent
- Train and Evaluate → PPO Trainer: optimizes policy
- Viewer User → Viewer App: opens viewer
- Viewer App → Scene Tree: renders scene (evidence: `vis_scripts/viser_m/src/viser/client/src/App.tsx`)
- Viewer App → WebSocket Messages: receives updates (evidence: `vis_scripts/viser_m/src/viser/client/src/App.tsx`)
- Viewer App → Browser Client: renders interface
- Scene Tree → Scene State: reads state
- Viewer App → GUI State: uses GUI state (evidence: `vis_scripts/viser_m/src/viser/client/src/App.tsx`)
- Splat Sort Worker → WASM Sorter: invokes sorter (evidence: `vis_scripts/viser/src/viser/client/src/Splatting/SplatSortWorker.ts`)
