# om-ai-lab/omagent architecture

> Architecture diagram of the GitHub repository om-ai-lab/omagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/om-ai-lab/omagent
- Repository: https://github.com/om-ai-lab/omagent

## Overview

##### High-Level System Type and Purpose
- The project is a full-fledged multimodal agent framework. It provides core libraries (omagent-core) that implement agent components and workflows (CoT, ReAct, SC-Cot, etc.) and a set of example applications (e.g., video understanding, VQA, mobile assistant).
- The README highlights key capabilities (flexible workflow orchestration, memory systems, integration with multimodal models such as VLM and video processing) and deployment patterns (including Docker configurations).

##### Main Architectural Components
a. Core Library Layer (omagent-core):
- This directory holds the core engine, advanced agent components, workflow orchestration modules, clients for various device interactions (CLI, web, app), memory modules (LTM and STM), and tool systems.
- Within omagent-core, you have submodules for engine (configuration, orchestration, HTTP APIs), advanced workflows (several folders for CoT, ToT, DNC, etc.), clients (devices for CLI, webpage, mobile), services (connectors like for Conductor, Milvus, Redis), and utilities.
b. Workflow and Agent Components:
- The “advanced_components/workflow” subdirectories implement the different reasoning and agent workflows (Chain-of-Thought, ReAct, etc.), and even include “agent” directories with further breakdown (e.g., input interfaces, output interfaces, and task-specific agents).
c. Deployment and Configuration:
- The Docker directory contains Docker Compose files and configuration files (such as container.yaml and bootstrap.yaml) that configure service dependencies, scaling, and runtime orchestration.
- The container.yaml and compile_container.py in each example illustrate how workflow definitions and dependency injection are managed.
d. Example Applications:
- Multiple “examples” subdirectories illustrate how to integrate different client interfaces (CLI, webpage, programmatic) with specific workflows (e.g., simple VQA, video understanding, outfit recommendation, mobile assistant).
- Each example comes with its own configuration files (configs/llms, configs/workers) that tailor the agent behaviors and model settings.
e. Documentation and Developer Support:
- The docs directory, along with the README and additional images, provides contextual and architectural insights explaining agent concepts, client integrations, memory systems, and container orchestration.

##### Relationships and Data Flow
- The core library (omagent-core) serves as the backend engine that handles task ingestion, workflow execution, and memory management.
- Client components (devices for CLI, web, mobile) send requests that are received by API endpoints or integrated directly within the agent workflow.
- The worker and orchestration modules interact with external connectors (e.g., Kafka for event queues, milvus/redis for memory storage) and the LLM models either via local deployment (Ollama, LocalAI) or remote API endpoints.
- Configuration files (like container.yaml) tie these components together, ensuring that dependencies (model configurations, task definitions, workflow orchestration settings) are injected at runtime.

##### Architectural Patterns
- Modular & Decoupled Design: Each component is isolated (agents, workflows, memory, connectors) permitting clear boundaries.
- Workflow Orchestration: Graph-based workflows manage tasks between components – shown as directional flows from input interfaces through processing and back to client interfaces.
- Containerization & Deployment Abstraction: Docker configurations allow replication of environments and easy scaling.
- Extensibility: The clearly separated “advanced_components,” “clients,” and “tool_system” layers allow plugging in new reasoning operators or client types without rewriting core logic.
- Multi-Layer Integration: There are clear layers for infrastructure (deployment, config), core business logic (engine, orchestration), and client presentation (CLI, webpage, app).

##### Final Recommendations
- Keep the architecture as detailed as possible by splitting the project into its numerous modules – from core engine to the example agents – so that every service and workflow is represented.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        client["Client Interfaces"]:::frontend
    end

    %% Backend Layer (omagent-core)
    subgraph "Backend Layer"
        core_lib["Agent Engine (Core Library)"]:::backend
        worker["Worker Orchestration"]:::backend
        adv_workflow["Workflow Orchestrator"]:::backend
        memory["Memory Modules (LTM/STM)"]:::backend
        tool["Tool System Modules"]:::backend
    end

    %% Deployment and Support
    subgraph "Deployment & Support"
        deploy["Deployment & Config"]:::deployment
        examples["Example Applications"]:::deployment
        docs["Documentation"]:::deployment
    end

    %% External Connectors
    ext["External Connectors (Kafka, Redis, Milvus)"]:::external

    %% Connections
    client -->|"sendRequest"| core_lib
    core_lib -->|"delegatesTo"| adv_workflow
    core_lib -->|"coordinates"| worker
    core_lib -->|"storesData"| memory
    core_lib -->|"utilizes"| tool
    core_lib -->|"integrates"| ext

    deploy -->|"configures"| core_lib
    examples -->|"demonstrates"| client
    docs --- deploy
    docs --- core_lib
    docs --- examples

    %% Styles
    classDef frontend fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef backend fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef deployment fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef external fill:#FFB6C1,stroke:#000,stroke-width:1px;

    %% Click Events
    click core_lib "https://github.com/om-ai-lab/omagent/tree/main/omagent-core/"
    click adv_workflow "https://github.com/om-ai-lab/omagent/tree/main/omagent-core/src/omagent_core/advanced_components/workflow/"
    click worker "https://github.com/om-ai-lab/omagent/tree/main/omagent-core/src/omagent_core/engine/automator/"
    click memory "https://github.com/om-ai-lab/omagent/tree/main/omagent-core/src/omagent_core/memories/"
    click client "https://github.com/om-ai-lab/omagent/tree/main/omagent-core/src/omagent_core/clients/devices/"
    click tool "https://github.com/om-ai-lab/omagent/tree/main/omagent-core/src/omagent_core/tool_system/"
    click deploy "https://github.com/om-ai-lab/omagent/tree/main/docker/"
    click examples "https://github.com/om-ai-lab/omagent/tree/main/examples/"
    click docs "https://github.com/om-ai-lab/omagent/tree/main/docs/"
```
