How omarmohamed5699/universal_manipulation_interface works
Project type & purpose
• This is a robotics research & deployment pipeline combining:
– Data collection (real and simulated)
– SLAM‐based preprocessing
– Dataset preparation (replay buffers)
– Training diffusion/RL policies
– Real‐world inference & evaluation
• It ships as a monolithic Python codebase, organized into two main libraries (“diffusion_policy” and “umi”), plus top‐level scripts and examples.
Top‐level components
A. Hardware & External Services
1. Robots & Grippers (UR5, Franka, WSG50)
2. Cameras (GoPro, Realsense, SpaceMouse input)
3. SLAM Service (ORB_SLAM3 via Docker)
B. Data Acquisition Layer
• “real_world” modules under both diffusion_policy/real_world and umi/real_world
• Scripts: scripts_slam_pipeline/, scripts/, scripts_real/
• Shared‐memory queue & ring‐buffer (diffusion_policy/shared_memory)
C. SLAM Pipeline
• run_slam_pipeline.py → calls umi/pipeline/aruco_detection.py + ORB_SLAM3
• Outputs raw trajectories and camera poses
D. Dataset Preparation
• scripts_slam_pipeline/07_generate_replay_buffer.py
• diffusion_policy/dataset/* (base_dataset, replay_buffer)
• Zarr/Replay‐buffer storage
E. Simulation Environments
• diffusion_policy/env/{block_pushing, kitchen, pusht, robomimic}
• env_runner: vectorized runners, wrappers (gym_util)
F. Model & Training Layer
1. Core library: diffusion_policy/model (common utils, vision, bet, diffusion)
2. Policies: diffusion_policy/policy/*.py
3. Workspaces: diffusion_policy/workspace/* (Hydra + accelerate)
4. train.py entrypoint
5. Dependencies: PyTorch, Hydra, Accelerate, timm, Minkowski, conda env
G. Model Zoo & Checkpoints
• Pretrained weights download/upload
• config files under diffusion_policy/config
H. Inference & Real‐world Deployment
• eval_real.py → loads checkpoint + real_world wrappers + robot drivers
• example/eval_robots_config.yaml
Interactions & data flows
1. Hardware → Real_World module
→ Shared_Memory → SLAM Pipeline
→ Raw Topic Data + Poses
2. SLAM Pipeline → Zarr dataset (replay buffer)
3. Simulation Envs ↔ Env_Runner → Replay Buffer (for offline training)
4. Dataset Module → DataLoader → Model (diffusion or bet)
5. Model + Policy → Training Loop → Checkpoint Store
6. Trained Checkpoint + Real_World modules → Inference → Robot Actuation
Architectural patterns & principles
• Modular layering: separation of concerns (HW interface / data processing / modeling / training / deployment)
• Config‐driven (Hydra): decouple code paths via YAML configs under diffusion_policy/config
• Shared‐memory for high‐throughput camera ↔ SLAM interop
• Vectorized environments for sample efficiency
• Pluggable policy heads (UNet, Transformer, BET)
Final checklist
– Call out key technologies: Python, PyTorch, Hydra, Docker, Zarr, Accelerate