# yanjieze/3d-diffusion-policy architecture

> Architecture diagram of the GitHub repository yanjieze/3d-diffusion-policy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yanjieze/3d-diffusion-policy
- Repository: https://github.com/yanjieze/3d-diffusion-policy

## Overview

##### Main system components
a. Data Ingestion & Storage
- Expert demonstrations (point clouds, images, depth, agent states/actions) stored under `3D-Diffusion-Policy/data/` or `$YOUR_REPO_PATH/data/*.zarr`.
- Config files (`.yaml`) under `diffusion_policy_3d/config/` and `config/task/`.
b. Environment Modules
- Sim wrappers under `diffusion_policy_3d/env/` (Adroit, DexArt, MetaWorld).
- Gym adapters under `diffusion_policy_3d/gym_util/`.
- Third-party simulator libraries: `third_party/VRL3`, `third_party/dexart-release`, `third_party/Metaworld`, `third_party/gym-0.21.0`, `third_party/mujoco-py`, etc.
c. Data Loading & Preprocessing
- Dataset classes under `diffusion_policy_3d/dataset/` (Adroit, DexArt, MetaWorld, RealDex).
- Replay buffer, sampler, point-cloud downsampling, cropping utilities in `common/`, `gym_util/`, and `scripts/convert_real_robot_data.py`.
d. Model & Policy
- Vision backbone: `model/vision/pointnet_extractor.py`.
- Diffusion policy: `model/diffusion/*` (conditional UNet1d, positional embeddings, EMA).
- Policy wrappers: `policy/base_policy.py`, `policy/dp3.py`, `policy/simple_dp3.py`.
e. Training & Evaluation
- Entry points: `train.py` and `eval.py` at repo root.
- Shell scripts under `scripts/` for generating demos, training (`train_policy.sh`), evaluation (`eval_policy.sh`), and GPU discovery.
- Logging to Weights & Biases (wandb) via `common/logger_util.py`.
- Checkpoint utilities in `common/checkpoint_util.py`.
f. Deployment & Visualization
- Real robot interface (Realsense, Franka, Allegro) described in README; conversion script for real data.
- Simple point-cloud visualizer under `visualizer/`.

##### Relationships & data flows
a. Raw demos (zarr, images, point clouds) → Preprocessing (cropping, FPS) → Dataset loader → Batches of (obs, actions).
b. Obs fed into Model sub-components: Vision (PointNet) → Feature tokens → Diffusion UNet1d → Action predictions.
c. Predicted actions → environment wrappers (sim or real robot) → next obs → optional closed-loop inference.
d. Training loop (PyTorch) manages sampling, loss computation, backprop, checkpointing.
e. wandb logging records metrics/visuals.
f. Scripts coordinate end-to-end flow: generation → training → evaluation.

##### Architectural patterns & design principles
- Modular layering:
– Config → Dataset → Model → Policy → Runner scripts
- Separation of concerns:
– “env” for sim interfacing, “dataset” for data loading, “model” for network definitions, “policy” for inference wrappers.
- Pluggable third-party backends:
– Hydra-like YAML configs to switch tasks (Adroit vs DexArt vs MetaWorld vs RealDex).
- Reusability:
– Common utilities for checkpointing, logging, PyTorch wrappers in `common/`.
- Extensibility:
– Adding a new task requires: env wrapper + runner + dataset + config file.

##### Detailed elements
- Data sources: raw demos (`.zarr`), expert policies under `third_party/VRL3/ckpts`, DexArt assets.
- Dataset classes & ReplayBuffer interacting with Data Layer.
- Vision model block (PointNet) feeding into Diffusion UNet.
- Policy wrapper applying noise schedules and sampling.
- Training orchestration:
– PyTorch DataLoader ← Dataset ← Preprocessor
– Optimizer/ Scheduler ← Model
– Loss calculation ← Diffusion
– Checkpoint Util & Logger Util
- Evaluation orchestration:
– eval.py → loads checkpoint → env_runner → policy → wandb/video.
- External services:
– wandb, Realsense camera, Franka/Allegro controllers.

##### Details
- Major sub-folders under `diffusion_policy_3d`: common, config, dataset, env, env_runner, gym_util, model, policy.
– Config files → Dataset loader (configurable hyperparams)
– Env wrappers ←→ Third-party simulator backends
– Model ← DataLoader ← Dataset ← Preprocessor
– Policy → EnvRunner → Sim/Real
– Scripts → orchestrate the above flows
- Do not omit the visualizer component (`visualizer/`), as it’s important for debugging.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        RawData(("Raw Demos")):::data
        ConfigFiles["Config Files"]:::core
    end

    %% Environment Layer
    subgraph "Environment Layer"
        EnvWrappers["Env Wrappers"]:::core
        GymAdapters["Gym Adapters"]:::core
        subgraph "Third-Party Simulators"
            VRL3["VRL3"]:::third
            DexArt["DexArt"]:::third
            Metaworld["Metaworld"]:::third
            GymLib["gym-0.21.0"]:::third
            MujocoPy["mujoco-py"]:::third
        end
    end

    %% ML Layer
    subgraph "ML Layer"
        Dataset["Dataset & Preprocessor"]:::core
        ReplayBuffer["Replay Buffer"]:::core
        Sampler["Sampler"]:::core
        CheckpointUtil["Checkpoint Util"]:::core
        LoggerUtil["Logger Util"]:::core
        PointNet["PointNet Extractor"]:::core
        DiffusionModel["Diffusion UNet1d + Components"]:::core
        subgraph "Policy Wrappers"
            BasePolicy["base_policy.py"]:::core
            DP3Policy["dp3.py"]:::core
            SimplePolicy["simple_dp3.py"]:::core
        end
    end

    %% Orchestration & Scripts
    subgraph "Orchestration & Scripts"
        TrainPy["train.py"]:::script
        EvalPy["eval.py"]:::script
        GenAdroit["gen_demonstration_adroit.sh"]:::script
        GenDexArt["gen_demonstration_dexart.sh"]:::script
        GenMeta["gen_demonstration_metaworld.sh"]:::script
        TrainSh["train_policy.sh"]:::script
        EvalSh["eval_policy.sh"]:::script
        FindGPU["find_gpu.sh"]:::script
        ConvertData["convert_real_robot_data.py"]:::script
    end

    %% Deployment & Visualization
    subgraph "Deployment & Visualization"
        SimReal["Simulators / Real Robot"]:::deploy
        Visualizer["PointCloud Visualizer"]:::deploy
        Checkpoints(("Checkpoints")):::data
        Wandb["Weights & Biases"]:::external
    end

    %% Data Flows
    RawData -->|load & preprocess| Dataset
    ConfigFiles -->|params| Dataset

    Dataset -->|batches| PointNet
    PointNet -->|features| DiffusionModel
    DiffusionModel -->|action preds| DP3Policy
    DP3Policy -->|apply actions| EnvWrappers
    EnvWrappers -->|calls| VRL3
    EnvWrappers -->|calls| DexArt
    EnvWrappers -->|calls| Metaworld
    EnvWrappers -->|calls| GymLib
    EnvWrappers -->|calls| MujocoPy

    %% Training Orchestration
    TrainPy -->|uses| Dataset
    TrainPy -->|uses| PointNet
    TrainPy -->|uses| DiffusionModel
    TrainPy -->|utils replay| ReplayBuffer
    TrainPy -->|utils sampler| Sampler
    TrainPy -->|checkpoint| CheckpointUtil
    CheckpointUtil -->|to| Checkpoints
    TrainPy -->|log| LoggerUtil
    LoggerUtil -->|to| Wandb

    %% Evaluation Orchestration
    EvalPy -->|load| Checkpoints
    EvalPy -->|run| DP3Policy
    EvalPy -->|env| SimReal
    EvalPy -->|log| LoggerUtil
    EvalPy -->|to| Wandb

    %% Scripts Flow
    GenAdroit -->|gen demos| RawData
    GenDexArt -->|gen demos| RawData
    GenMeta -->|gen demos| RawData
    TrainSh -->|invoke| TrainPy
    EvalSh -->|invoke| EvalPy
    FindGPU -->|discover| TrainPy
    ConvertData -->|convert| RawData

    %% Visualization
    DP3Policy -->|visualize| Visualizer
    SimReal -->|visualize| Visualizer

    %% Click Events
    click ConfigFiles "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/diffusion_policy_3d/config/"
    click EnvWrappers "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/diffusion_policy_3d/env/"
    click GymAdapters "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/diffusion_policy_3d/gym_util/"
    click VRL3 "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/third_party/VRL3/"
    click DexArt "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/third_party/dexart-release/"
    click Metaworld "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/third_party/Metaworld/"
    click GymLib "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/third_party/gym-0.21.0/"
    click MujocoPy "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/"
    click Dataset "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/diffusion_policy_3d/dataset/"
    click ReplayBuffer "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/common/replay_buffer.py"
    click Sampler "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py"
    click CheckpointUtil "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/common/checkpoint_util.py"
    click LoggerUtil "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/common/logger_util.py"
    click PointNet "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/model/vision/pointnet_extractor.py"
    click DiffusionModel "https://github.com/yanjieze/3d-diffusion-policy/tree/master/3D-Diffusion-Policy/diffusion_policy_3d/model/diffusion/"
    click BasePolicy "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/policy/base_policy.py"
    click DP3Policy "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/policy/dp3.py"
    click SimplePolicy "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/diffusion_policy_3d/policy/simple_dp3.py"
    click TrainPy "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/train.py"
    click EvalPy "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/eval.py"
    click GenAdroit "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/gen_demonstration_adroit.sh"
    click GenDexArt "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/gen_demonstration_dexart.sh"
    click GenMeta "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/gen_demonstration_metaworld.sh"
    click TrainSh "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/train_policy.sh"
    click EvalSh "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/eval_policy.sh"
    click FindGPU "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/find_gpu.sh"
    click ConvertData "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/scripts/convert_real_robot_data.py"
    click Visualizer "https://github.com/yanjieze/3d-diffusion-policy/blob/master/3D-Diffusion-Policy/visualizer/visualizer/pointcloud.py"

    %% Styles
    classDef core fill:#bbdefb,stroke:#2196f3,color:#0d47a1;
    classDef third fill:#c8e6c9,stroke:#388e3c,color:#1b5e20;
    classDef data fill:#ffe0b2,stroke:#fb8c00,color:#e65100;
    classDef script fill:#f3e5f5,stroke:#9c27b0,color:#4a148c;
    classDef deploy fill:#eeeeee,stroke:#9e9e9e,color:#424242;
    classDef external fill:#ffcdd2,stroke:#e53935,color:#b71c1c;
```
