How openmind/om1 works
Project type and purpose
• OM1 is a modular, Python-based AI runtime rather than a traditional web or mobile full-stack. Its purpose is to orchestrate multimodal inputs (camera, LIDAR, web, social media), feed them through fusion and LLM pipelines, and emit high-level “actions” to robots or simulations. It is not a CRUD app or simple library; it’s a pipeline-driven, plugin-based middleware for AI agents.
File structure
• Top-level directories of interest:
• config/ ← JSON5 configs for agents, inputs, LLMs, actions
• src/runtime/ ← bootstrapping, config loading, logging, robotics setup
• src/inputs/ ← input-stage plugins and orchestrator
• src/backgrounds/ ← background sensing/state updates
• src/fuser/ ← sensor fusion components
• src/llm/ ← LLM abstraction plugins
• src/actions/ ← high-level action interfaces + connectors to hardware or simulators
• src/providers/ ← low-level providers: ROS2 publishers, Zenoh, drivers, third-party APIs
• src/simulators/ ← WebSim, Gazebo connectors
• run.py ← entry point that ties everything together
• Build scripts or CI: pyproject.toml, GitHub Actions workflows, Dockerfiles under gazebo/.
README for additional insights
• It calls out core modules: inputs, actions, backgrounds, LLMs, providers, simulators.
• Mentions Zenoh/ROS2/CycloneDDS as middleware.
• Blue = Core runtime modules (runtime, orchestrator, fuser)
• Green = Plugin types (inputs, backgrounds, llm, actions, simulators)
• Orange = External middleware/services (Zenoh, ROS2, OpenAI)
• Gray = Hardware / Dockerized simulators
• Clouds for external APIs and hardware SDKs
• Stencils for provider abstractions
Key points to capture in extreme detail
• How config files drive plugin loading
• Explicitly call out all plugin categories
• WebSim on the side subscribing to bus for real-time metrics