How real-stanford/diffusion_policy works
Project Type and Purpose
• The project is a robotics control framework using diffusion and transformer-based policies. It supports multiple tasks (e.g., block pushing, kitchen manipulation, and push-T) and accommodates both simulation and real-world scenarios.
• The README details both simulation and real robot operation, including special hardware requirements (notably a 3Dconnexion SpaceMouse for teleoperation).
Core Components from the File Structure
• Configurations: The “diffusion_policy/config” directory contains YAML files to configure both tasks (e.g., different variants of block pushing, kitchen tasks) and training pipelines (e.g., diffusion transformers, unet methods).
• Datasets and Replay Buffers: Within “diffusion_policy/dataset” you see dataset wrapper classes that load and preprocess demonstration data, including normalization routines.
• Environment Interfaces: Under “diffusion_policy/env” and “diffusion_policy/env_runner”, various environment wrappers exist for simulation (gym-based environments) and real-world experiments.
• Workspace & Training: In “diffusion_policy/workspace” the logic for launching training experiments is encapsulated. The “train.py” script brings these pieces together using Hydra for dynamic configuration.
• Shared Memory & Real-World Integration: The “diffusion_policy/shared_memory” and “diffusion_policy/real_world” directories incorporate modules for efficient data streaming (such as using SharedMemoryRingBuffer) from cameras and sensors. Within “diffusion_policy/real_world” you find “spacemouse.py” and “spacemouse_shared_memory.py”—key modules for mouse teleoperation control.
• External Data and Logging: Files like “multirun_metrics.py”, wandb integration in training scripts, and output directories show where logs, metrics, and checkpoints are managed.
Smart Mouse Control Module
• In the real-robot control chain, the SpaceMouse is used for teleoperation. The system interacts with the SpaceMouse via:
• The “spacemouse.py” module: This component likely reads raw inputs from the physical device.
• The “spacemouse_shared_memory.py” implementation: It passes the teleop commands via shared memory interfaces to ensure minimal latency.
• Smart Mouse Control module to the “RealEnv” component (or “Real Robot” wrapper) and further into the policy execution pipeline. For example, show bidirectional data flow: user inputs from the SpaceMouse trigger teleoperated motion commands which are incorporated into the policy’s real-time planning loop.