# openmind/om1 architecture

> Architecture diagram of the GitHub repository openmind/om1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openmind/om1
- Repository: https://github.com/openmind/om1

## Overview

##### Project type and purpose
- OM1 is a modular, Python-based AI runtime rather than a traditional web or mobile full-stack. Its purpose is to orchestrate multimodal inputs (camera, LIDAR, web, social media), feed them through fusion and LLM pipelines, and emit high-level “actions” to robots or simulations. It is not a CRUD app or simple library; it’s a pipeline-driven, plugin-based middleware for AI agents.

##### File structure
- Top-level directories of interest:
- config/            ← JSON5 configs for agents, inputs, LLMs, actions
- src/runtime/       ← bootstrapping, config loading, logging, robotics setup
- src/inputs/        ← input-stage plugins and orchestrator
- src/backgrounds/   ← background sensing/state updates
- src/fuser/         ← sensor fusion components
- src/llm/           ← LLM abstraction plugins
- src/actions/       ← high-level action interfaces + connectors to hardware or simulators
- src/providers/     ← low-level providers: ROS2 publishers, Zenoh, drivers, third-party APIs
- src/simulators/    ← WebSim, Gazebo connectors
- run.py             ← entry point that ties everything together
- Build scripts or CI: pyproject.toml, GitHub Actions workflows, Dockerfiles under gazebo/.

##### README for additional insights
- It calls out core modules: inputs, actions, backgrounds, LLMs, providers, simulators.
- Mentions Zenoh/ROS2/CycloneDDS as middleware.

- Blue = Core runtime modules (runtime, orchestrator, fuser)
- Green = Plugin types (inputs, backgrounds, llm, actions, simulators)
- Orange = External middleware/services (Zenoh, ROS2, OpenAI)
- Gray = Hardware / Dockerized simulators
- Clouds for external APIs and hardware SDKs
- Stencils for provider abstractions

##### Key points to capture in extreme detail
- How config files drive plugin loading
- Explicitly call out all plugin categories
- WebSim on the side subscribing to bus for real-time metrics

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Runtime
    subgraph "Core Runtime"
        direction TB
        Run["run.py"]:::core
        ConfigLoader["Config Loader"]:::core
        Cortex["Cortex (Orchestrator)"]:::core
        Logging["Logging"]:::core
        Robotics["Robotics Setup"]:::core
    end

    %% Input Pipeline
    subgraph "Input Pipeline"
        direction TB
        InputOrch["Input Orchestrator"]:::plugin
        BaseLoop["Base Loop"]:::plugin
        InputPlugins["Input Plugins"]:::plugin
    end

    %% Background Updaters
    subgraph "Background State Updaters"
        direction TB
        BgOrch["Background Orchestrator"]:::plugin
        BgBase["Background Base"]:::plugin
        BgPlugins["Background Plugins"]:::plugin
    end

    %% Sensor Fusion
    subgraph "Sensor Fusion"
        direction TB
        FusionEngine["Fusion Engine"]:::core
    end

    %% LLM & VLM Layer
    subgraph "LLM & VLM Layer"
        direction TB
        LLMLayer["LLM Abstraction"]:::plugin
        LLPlugins["LLM Plugins"]:::plugin
    end

    %% Orchestration Core (Actions)
    subgraph "Action Orchestrator"
        direction TB
        ActionOrch["Actions Orchestrator"]:::core
        ActionBase["Actions Base"]:::core
        ActionInterfaces["Action Interfaces & Connectors"]:::plugin
    end

    %% Providers & Middleware
    subgraph "Providers & Middleware Bus"
        direction TB
        ProvidersBus["Zenoh / ROS2 / CycloneDDS"]:::middleware
    end

    %% Simulators & Web UI
    subgraph "Simulators & Web UI"
        direction TB
        SimOrch["Simulators Orchestrator"]:::plugin
        SimBase["Simulators Base"]:::plugin
        WebSim["WebSim Plugin"]:::plugin
    end

    %% External Services
    subgraph "External Services"
        direction TB
        ExternalAPIs["OpenAI, ElevenLabs, Social APIs"]:::middleware
    end

    %% Hardware SDKs & Gazebo
    subgraph "Hardware & Simulated Environments"
        direction TB
        HardwareSDKs["Unitree, Tesla, Arduino SDKs"]:::hw
        GazeboDocker["Gazebo / Docker"]:::hw
    end

    %% Connections
    Run --> ConfigLoader
    ConfigLoader --> Cortex
    Cortex --> Logging
    Cortex --> Robotics

    %% Data Ingestion
    ProvidersBus --> InputOrch
    ProvidersBus --> BgOrch

    InputOrch --> BaseLoop
    BaseLoop --> InputPlugins
    BgOrch --> BgBase
    BgBase --> BgPlugins

    InputOrch --> FusionEngine
    BgOrch --> FusionEngine

    FusionEngine --> Cortex

    Cortex --> LLMLayer
    LLMLayer --> LLPlugins
    LLMLayer --> ExternalAPIs

    Cortex --> ActionOrch
    ActionOrch --> ActionBase
    ActionBase --> ActionInterfaces
    ActionInterfaces --> ProvidersBus

    ProvidersBus --> HardwareSDKs
    ProvidersBus --> GazeboDocker
    ProvidersBus --> SimOrch

    SimOrch --> SimBase
    SimBase --> WebSim

    WebSim --> ProvidersBus

    %% Click Events
    click ConfigLoader "https://github.com/openmind/om1/blob/main/src/runtime/config.py"
    click Run "https://github.com/openmind/om1/blob/main/run.py"
    click Cortex "https://github.com/openmind/om1/blob/main/src/runtime/cortex.py"
    click Logging "https://github.com/openmind/om1/blob/main/src/runtime/logging.py"
    click Robotics "https://github.com/openmind/om1/blob/main/src/runtime/robotics.py"
    click InputOrch "https://github.com/openmind/om1/blob/main/src/inputs/orchestrator.py"
    click BaseLoop "https://github.com/openmind/om1/blob/main/src/inputs/base/loop.py"
    click InputPlugins "https://github.com/openmind/om1/tree/main/src/inputs/plugins/"
    click BgOrch "https://github.com/openmind/om1/blob/main/src/backgrounds/orchestrator.py"
    click BgBase "https://github.com/openmind/om1/blob/main/src/backgrounds/base.py"
    click BgPlugins "https://github.com/openmind/om1/tree/main/src/backgrounds/plugins/"
    click FusionEngine "https://github.com/openmind/om1/blob/main/src/fuser/__init__.py"
    click ActionOrch "https://github.com/openmind/om1/blob/main/src/actions/orchestrator.py"
    click ActionBase "https://github.com/openmind/om1/blob/main/src/actions/base.py"
    click ActionInterfaces "https://github.com/openmind/om1/blob/main/src/actions/*/interface.py"
    click LLMLayer "https://github.com/openmind/om1/blob/main/src/llm/output_model.py"
    click LLPlugins "https://github.com/openmind/om1/tree/main/src/llm/plugins/"
    click ProvidersBus "https://github.com/openmind/om1/tree/main/src/providers/"
    click SimOrch "https://github.com/openmind/om1/blob/main/src/simulators/orchestrator.py"
    click SimBase "https://github.com/openmind/om1/blob/main/src/simulators/base.py"
    click WebSim "https://github.com/openmind/om1/blob/main/src/simulators/plugins/WebSim.py"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef plugin fill:#D0FFD6,stroke:#333,stroke-width:1px
    classDef middleware fill:#FFD8A8,stroke:#333,stroke-width:1px
    classDef hw fill:#E0E0E0,stroke:#333,stroke-width:1px
```
