# omarmohamed5699/universal_manipulation_interface architecture

> Architecture diagram of the GitHub repository omarmohamed5699/universal_manipulation_interface, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/omarmohamed5699/universal_manipulation_interface
- Repository: https://github.com/omarmohamed5699/universal_manipulation_interface

## Overview

##### Project type & purpose
- This is a robotics research & deployment pipeline combining:
– Data collection (real and simulated)
– SLAM‐based preprocessing
– Dataset preparation (replay buffers)
– Training diffusion/RL policies
– Real‐world inference & evaluation
- It ships as a monolithic Python codebase, organized into two main libraries (“diffusion_policy” and “umi”), plus top‐level scripts and examples.

##### Top‐level components
A. Hardware & External Services
1. Robots & Grippers (UR5, Franka, WSG50)
2. Cameras (GoPro, Realsense, SpaceMouse input)
3. SLAM Service (ORB_SLAM3 via Docker)
B. Data Acquisition Layer
- “real_world” modules under both diffusion_policy/real_world and umi/real_world
- Scripts: scripts_slam_pipeline/, scripts/, scripts_real/
- Shared‐memory queue & ring‐buffer (diffusion_policy/shared_memory)
C. SLAM Pipeline
- run_slam_pipeline.py → calls umi/pipeline/aruco_detection.py + ORB_SLAM3
- Outputs raw trajectories and camera poses
D. Dataset Preparation
- scripts_slam_pipeline/07_generate_replay_buffer.py
- diffusion_policy/dataset/* (base_dataset, replay_buffer)
- Zarr/Replay‐buffer storage
E. Simulation Environments
- diffusion_policy/env/{block_pushing, kitchen, pusht, robomimic}
- env_runner: vectorized runners, wrappers (gym_util)
F. Model & Training Layer
1. Core library: diffusion_policy/model (common utils, vision, bet, diffusion)
2. Policies: diffusion_policy/policy/*.py
3. Workspaces: diffusion_policy/workspace/* (Hydra + accelerate)
4. train.py entrypoint
5. Dependencies: PyTorch, Hydra, Accelerate, timm, Minkowski, conda env
G. Model Zoo & Checkpoints
- Pretrained weights download/upload
- config files under diffusion_policy/config
H. Inference & Real‐world Deployment
- eval_real.py → loads checkpoint + real_world wrappers + robot drivers
- example/eval_robots_config.yaml

##### Interactions & data flows
1. Hardware → Real_World module
→ Shared_Memory → SLAM Pipeline
→ Raw Topic Data + Poses
2. SLAM Pipeline → Zarr dataset (replay buffer)
3. Simulation Envs ↔ Env_Runner → Replay Buffer (for offline training)
4. Dataset Module → DataLoader → Model (diffusion or bet)
5. Model + Policy → Training Loop → Checkpoint Store
6. Trained Checkpoint + Real_World modules → Inference → Robot Actuation

##### Architectural patterns & principles
- Modular layering: separation of concerns (HW interface / data processing / modeling / training / deployment)
- Config‐driven (Hydra): decouple code paths via YAML configs under diffusion_policy/config
- Shared‐memory for high‐throughput camera ↔ SLAM interop
- Vectorized environments for sample efficiency
- Pluggable policy heads (UNet, Transformer, BET)

##### Final checklist
– Call out key technologies: Python, PyTorch, Hydra, Docker, Zarr, Accelerate

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Hardware & External Services
    subgraph "Hardware & External Services"
        direction TB
        subgraph "Robots & Grippers"
            direction TB
            Fr("franka_interpolation_controller.py"):::hardware
            W1("wsg_controller.py"):::hardware
            W2("wsg_binary_driver.py"):::hardware
        end
        subgraph "Cameras & Input Devices"
            direction TB
            C1("uvc_camera.py"):::hardware
            C2("multi_uvc_camera.py"):::hardware
            C3("single_realsense.py"):::hardware
            C4("multi_realsense.py"):::hardware
            C5("spacemouse.py"):::hardware
            C6("spacemouse_shared_memory.py"):::hardware
        end
        subgraph "SLAM Service"
            direction TB
            ORB("orb_slam_util.py"):::hardware
            RunSLAM("run_slam_pipeline.py"):::hardware
        end
    end

    %% Data Acquisition & SLAM
    subgraph "Data Acquisition & SLAM"
        direction TB
        UMI_RW("umi/real_world/*"):::data
        ML_RW("diffusion_policy/real_world/*"):::data
        SMQ1("shared_memory_queue.py\n(umi)"):::data
        SMQ2("shared_memory_queue.py\n(diffusion_policy)"):::data
        RBUF1("shared_memory_ring_buffer.py\n(umi)"):::data
        RBUF2("shared_memory_ring_buffer.py\n(diffusion_policy)"):::data
        Aru("aruco_detection.py"):::data
    end

    %% Dataset Preparation & Simulation
    subgraph "Dataset & Simulation"
        direction TB
        subgraph DatasetPrep
            direction TB
            GenRB("07_generate_replay_buffer.py"):::data
            BaseDS("base_dataset.py"):::data
            RB("replay_buffer.py"):::data
            ZARR("Zarr/Replay-buffer"):::data
        end
        subgraph Simulation Envs
            direction TB
            BP("block_pushing/"):::data
            KT("kitchen/"):::data
            PT("pusht/"):::data
            RM("robomimic/"):::data
            ER("env_runner/"):::data
            GU("gym_util/"):::data
        end
    end

    %% Model & Training
    subgraph "Model & Training"
        direction TB
        Core("diffusion_policy/model/"):::model
        Pol("diffusion_policy/policy/"):::model
        WS("diffusion_policy/workspace/"):::model
        Train("train.py"):::model
    end

    %% Model Zoo & Config
    subgraph "Model Zoo & Config"
        direction TB
        CFG("diffusion_policy/config/"):::config
        CKPT("checkpoint_util.py"):::model
    end

    %% Inference & Deployment
    subgraph "Inference & Deployment"
        direction TB
        Eval("eval_real.py"):::model
        RIU("real_inference_util.py"):::model
        RC("example/eval_robots_config.yaml"):::config
        SCR("scripts_real/"):::data
    end

    %% Data Flows
    Fr -->|sensor stream| UMI_RW
    W1 --> UMI_RW
    W2 --> UMI_RW
    C1 --> UMI_RW
    C2 --> UMI_RW
    C3 --> ML_RW
    C4 --> ML_RW
    C5 --> ML_RW
    C6 --> ML_RW

    UMI_RW -->|shared-memory| SMQ1
    ML_RW -->|shared-memory| SMQ2
    SMQ1 --> RBUF1
    SMQ2 --> RBUF2
    RBUF1 --> Aru
    RBUF2 --> Aru
    RunSLAM --> Aru

    Aru -->|raw trajectories| GenRB
    GenRB --> BaseDS
    GenRB --> RB
    RB --> ZARR
    ZARR --> BaseDS

    BP --> ER
    KT --> ER
    PT --> ER
    RM --> ER
    ER --> RB
    GU --> ER

    BaseDS --> Core
    Core --> Pol
    Pol --> Train
    WS --> Train

    Train --> CKPT
    CKPT --> Eval
    CKPT --> CFG
    CFG --> Train

    Eval --> RIU
    Eval --> RC
    RIU --> Fr
    RIU --> W1
    SCR --> Eval

    %% Click Events
    click Fr "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/real_world/franka_interpolation_controller.py"
    click W1 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/real_world/wsg_controller.py"
    click W2 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/real_world/wsg_binary_driver.py"
    click C1 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/real_world/uvc_camera.py"
    click C2 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/real_world/multi_uvc_camera.py"
    click C3 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/real_world/single_realsense.py"
    click C4 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/real_world/multi_realsense.py"
    click C5 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/real_world/spacemouse.py"
    click C6 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/real_world/spacemouse_shared_memory.py"
    click ORB "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/common/orb_slam_util.py"
    click RunSLAM "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/run_slam_pipeline.py"
    click UMI_RW "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/umi/real_world/"
    click ML_RW "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/real_world/"
    click SMQ1 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/shared_memory/shared_memory_queue.py"
    click SMQ2 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/shared_memory/shared_memory_queue.py"
    click RBUF1 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/shared_memory/shared_memory_ring_buffer.py"
    click RBUF2 "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/shared_memory/shared_memory_ring_buffer.py"
    click Aru "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/umi/pipeline/aruco_detection.py"
    click GenRB "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/scripts_slam_pipeline/07_generate_replay_buffer.py"
    click BaseDS "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/dataset/base_dataset.py"
    click RB "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/common/replay_buffer.py"
    click BP "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/env/block_pushing/"
    click KT "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/env/kitchen/"
    click PT "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/env/pusht/"
    click RM "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/env/robomimic/"
    click ER "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/env_runner/"
    click GU "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/gym_util/"
    click Core "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/model/"
    click Pol "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/policy/"
    click WS "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/workspace/"
    click Train "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/train.py"
    click CFG "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/diffusion_policy/config/"
    click CKPT "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/common/checkpoint_util.py"
    click Eval "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/eval_real.py"
    click RIU "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/diffusion_policy/real_world/real_inference_util.py"
    click RC "https://github.com/omarmohamed5699/universal_manipulation_interface/blob/main/example/eval_robots_config.yaml"
    click SCR "https://github.com/omarmohamed5699/universal_manipulation_interface/tree/main/scripts_real/"

    %% Styles
    classDef hardware fill:#dddddd,stroke:#333,stroke-width:1px
    classDef data fill:#b3d9ff,stroke:#333,stroke-width:1px
    classDef model fill:#b3ffb3,stroke:#333,stroke-width:1px
    classDef config fill:#fff2b3,stroke:#333,stroke-width:1px
```
