How yanjieze/3d-diffusion-policy works
Main system components
a. Data Ingestion & Storage
• Expert demonstrations (point clouds, images, depth, agent states/actions) stored under 3D-Diffusion-Policy/data/ or $YOUR_REPO_PATH/data/*.zarr.
• Config files (.yaml) under diffusion_policy_3d/config/ and config/task/.
b. Environment Modules
• Sim wrappers under diffusion_policy_3d/env/ (Adroit, DexArt, MetaWorld).
• Gym adapters under diffusion_policy_3d/gym_util/.
• Third-party simulator libraries: third_party/VRL3, third_party/dexart-release, third_party/Metaworld, third_party/gym-0.21.0, third_party/mujoco-py, etc.
c. Data Loading & Preprocessing
• Dataset classes under diffusion_policy_3d/dataset/ (Adroit, DexArt, MetaWorld, RealDex).
• Replay buffer, sampler, point-cloud downsampling, cropping utilities in common/, gym_util/, and scripts/convert_real_robot_data.py.
d. Model & Policy
• Vision backbone: model/vision/pointnet_extractor.py.
• Diffusion policy: model/diffusion/* (conditional UNet1d, positional embeddings, EMA).
• Policy wrappers: policy/base_policy.py, policy/dp3.py, policy/simple_dp3.py.
e. Training & Evaluation
• Entry points: train.py and eval.py at repo root.
• Shell scripts under scripts/ for generating demos, training (train_policy.sh), evaluation (eval_policy.sh), and GPU discovery.
• Logging to Weights & Biases (wandb) via common/logger_util.py.
• Checkpoint utilities in common/checkpoint_util.py.
f. Deployment & Visualization
• Real robot interface (Realsense, Franka, Allegro) described in README; conversion script for real data.
• Simple point-cloud visualizer under visualizer/.
Relationships & data flows
a. Raw demos (zarr, images, point clouds) → Preprocessing (cropping, FPS) → Dataset loader → Batches of (obs, actions).
b. Obs fed into Model sub-components: Vision (PointNet) → Feature tokens → Diffusion UNet1d → Action predictions.
c. Predicted actions → environment wrappers (sim or real robot) → next obs → optional closed-loop inference.
d. Training loop (PyTorch) manages sampling, loss computation, backprop, checkpointing.
e. wandb logging records metrics/visuals.
f. Scripts coordinate end-to-end flow: generation → training → evaluation.
Architectural patterns & design principles
• Modular layering:
– Config → Dataset → Model → Policy → Runner scripts
• Separation of concerns:
– “env” for sim interfacing, “dataset” for data loading, “model” for network definitions, “policy” for inference wrappers.
• Pluggable third-party backends:
– Hydra-like YAML configs to switch tasks (Adroit vs DexArt vs MetaWorld vs RealDex).
• Reusability:
– Common utilities for checkpointing, logging, PyTorch wrappers in common/.
• Extensibility:
– Adding a new task requires: env wrapper + runner + dataset + config file.
Detailed elements
• Data sources: raw demos (.zarr), expert policies under third_party/VRL3/ckpts, DexArt assets.
• Dataset classes & ReplayBuffer interacting with Data Layer.
• Vision model block (PointNet) feeding into Diffusion UNet.
• Policy wrapper applying noise schedules and sampling.
• Training orchestration:
– PyTorch DataLoader ← Dataset ← Preprocessor
– Optimizer/ Scheduler ← Model
– Loss calculation ← Diffusion
– Checkpoint Util & Logger Util
• Evaluation orchestration:
– eval.py → loads checkpoint → env_runner → policy → wandb/video.
• External services:
– wandb, Realsense camera, Franka/Allegro controllers.
Details
• Major sub-folders under diffusion_policy_3d: common, config, dataset, env, env_runner, gym_util, model, policy.
– Config files → Dataset loader (configurable hyperparams)
– Env wrappers ←→ Third-party simulator backends
– Model ← DataLoader ← Dataset ← Preprocessor
– Policy → EnvRunner → Sim/Real
– Scripts → orchestrate the above flows
• Do not omit the visualizer component (visualizer/), as it’s important for debugging.