How alanyoung123/physworld works
Project type and purpose
• PhysWorld is an open-source research framework for learning physics-consistent world models of deformable objects from real video. It combines a simulator-based data synthesis pipeline with a graph neural network (GNN) world model for fast, accurate future state prediction.
Main architectural components
a. Video Input & Preprocessing
• Ingest raw video frames (RGB or depth)
• Preprocess for segmentation, object tracking, and mesh/voxel extraction
b. Physics-Aware Digital Twin Constructor
• Constitutive Model Selector (chooses material models in MPM)
• Global-to-Local Physical Property Optimizer (fits simulator parameters to real video)
c. Demonstration Synthesis Pipeline
• Part-Aware Perturbation Module (perturbs local properties to generate diverse behaviors)
• MPM Simulator (runs forward simulations, renders 4D trajectories)
d. World Model Training
• Dataset Assembler (packages synthetic demonstrations)
• GNN-Based Predictor (implements node/edge features, message passing in PyTorch/PyG)
• Training Engine (handles loss functions, batching, early stopping)
e. Real-Video Refinement Loop
• Refinement Optimizer (fine-tunes physical properties using real video loops)
• Data Fusion Layer (melds real and synthetic data)
f. Inference Engine & Export
• Fast GNN Inference Module (performs rollouts at ~47× speedup)
• API/CLI Interface (exposes model for downstream tasks in robotics/VR/AR)
g. Supporting Infrastructure
• Data Storage (synthetic trajectories, trained checkpoints)
• Configuration & Experiment Scripts (hyperparameters, job submission)
• Visualization & Logging (TensorBoard, matplotlib figures under /figures)
Architectural patterns & design principles
• Pipeline Pattern: clear stages from data ingestion to inference
• Modular Separation: simulation, data synthesis, ML training, and refinement are decoupled
• Plugin/Extension Points: constitutive model selector and perturbation modules can be extended
• Data-Driven Simulation: tight loop between real data calibration and physics simulator
Relevant technologies & libraries
• MPM Simulator (e.g., custom C++/CUDA code or Taichi MPM)
• PyTorch & PyTorch Geometric for GNN implementation
• Optimization libraries (e.g., SciPy for global-to-local fitting)
• Experiment management (Hydra, YAML config files)
• Visualization (matplotlib, TensorBoard)