# shawnvirtly/omniparser architecture

> Architecture diagram of the GitHub repository shawnvirtly/omniparser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shawnvirtly/omniparser
- Repository: https://github.com/shawnvirtly/omniparser

## Overview

##### Project Type and Purpose
- OmniParser is primarily a computer vision and LLM integration tool focused on parsing screenshots into structured GUI elements. This underpins its role as a “Screen Parsing tool for Pure Vision Based GUI Agent” that also provides control over a Windows 11 VM (see OmniTool’s “omnibox” component).
- The project serves both as a research/demo tool (via Gradio demo and Jupyter notebook examples) and as a full-fledged backend service (via the parser server and VM control scripts).

##### File Structure and Key Components
- The top-level directories and files indicate several key functional areas:
- Root-level files (e.g., README.md, LICENSE, SECURITY.md, demo.ipynb) provide overall project documentation, licensing, and demonstration examples.
- The “omnitool” directory is central and further divided into:
- The “agent” folder (with submodules like anthropic_agent.py, vlm_agent.py, and llm_utils libraries) responsible for interfacing with different LLM services.
- The “executor” folder (e.g., anthropic_executor.py) that executes the actions generated by the agents.
- “tools” which likely offer additional support functions (like computer and screen_capture utilities).
- “loop.py” may be managing asynchronous or iterative workflows.
▸ omnibox – Provides components for managing a Windows 11 virtual machine, including a Dockerfile, compose file, and various VM setup and management scripts. This part is key for the “control a Windows 11 VM” functionality.
▸ omniparserserver – Contains the standalone server module (omniparserserver.py) that probably wraps the parsing logic into an API or service.
- The “util” directory holds general utility libraries (e.g., omniParser, box_annotator, utils) that underpin core parsing functionality and image analysis.
- Other folders like “docs” and “eval” provide supporting documentation and evaluation scripts/logs.

##### Architectural Components and Their Relationships
a. Frontend/Interactive UI:
- Gradio demo (omnitool/gradio/app.py) serves as the user interface layer for running interactive demos.
- This is the entry point for users to input screenshots and view parsed, structured outputs.

b. Agent and Executor Layers:
- Under “omnitool/gradio/agent” are modules dedicated to interfacing with LLMs (Anthropic, VLM-based, etc.). They support processing user inputs and calling external APIs via utilities found under “llm_utils.”
- The “executor” modules (e.g., anthropic_executor.py) take care of executing commands or actions based on parsed information.

c. Backend Parsing Service:
- OmniParser’s core parsing logic lives in “util” and is integrated into the “omniparserserver.” This service is responsible for parsing screenshots into structural elements.
- It can be depicted as a central processing unit that handles image analysis, GUI element detection, and annotation generation.

d. VM Control & Deployment:
- The “omnitool/omnibox” component is dedicated to managing Windows 11 VMs. It includes Docker and compose files as well as scripts for VM operation (e.g., manage_vm shell and PowerShell scripts).
- This as an infrastructure or deployment service that interfaces with the parsed actions – translating commands into VM control operations.

e. External Dependencies:

##### Key Architectural Patterns and Design Principles
- Layered Architecture: Clearly separate UI (Gradio), Application Logic (agents and executors), and Core Parsing Services.
- Microservice or Modular Pattern: The OmniTool is split into distinct submodules (gradio-based UI, omnibox for VM control, and a dedicated server for omniparsing), stressing decoupling.
- Integration of External Services: Highlight how the agents use “llm_utils” to interact with various LLM APIs.
- Infrastructure as Code: Note the inclusion of Docker and compose files for containerizing and managing parts of the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Frontend Layer"
        UI["Gradio Demo Interface"]:::ui
    end

    subgraph "Agent & Executor Layer"
        A1["Anthropic Agent"]:::agent
        A2["VLM Agent"]:::agent
        A3["LLM Utilities"]:::agent
        EXE["Anthropic Executor"]:::agent
        TSF["Tool Support Functions"]:::agent
        LOOP["Workflow Loop"]:::agent
    end

    subgraph "Backend Parsing Layer"
        BP["Backend Parsing Service"]:::parser
        CPU["Core Parsing Utilities"]:::parser
    end

    subgraph "VM Control & Deployment Layer"
        VMControl["VM Control & Deployment"]:::vm
        VMBuild["VM Build & Setup"]:::vm
    end

    subgraph "External Services"
        EXT1["External LLM Services"]:::external
        EXT2["External Model Repositories"]:::external
    end

    %% Connections from UI to Agent Modules
    UI -->|"screenshot"| A1
    UI -->|"screenshot"| A2

    %% Agent Modules interactions
    A1 -->|"uses"| A3
    A2 -->|"uses"| A3
    A3 -->|"process_request"| BP
    BP -->|"utilizes"| CPU
    CPU -->|"provides_result"| EXE
    EXE -->|"triggers_VM_actions"| VMControl

    %% Support and workflow modules
    TSF -->|"supports"| A1
    TSF -->|"supports"| EXE
    LOOP -->|"manages"| A1
    LOOP -->|"manages"| EXE

    %% External integrations from LLM Utilities
    A3 -->|"calls"| EXT1
    A3 -->|"retrieves"| EXT2

    %% VM Control integration
    VMControl -->|"configures"| VMBuild

    %% Click Events
    click UI "https://github.com/shawnvirtly/omniparser/blob/master/omnitool/gradio/app.py"
    click A1 "https://github.com/shawnvirtly/omniparser/blob/master/omnitool/gradio/agent/anthropic_agent.py"
    click A2 "https://github.com/shawnvirtly/omniparser/blob/master/omnitool/gradio/agent/vlm_agent.py"
    click A3 "https://github.com/shawnvirtly/omniparser/tree/master/omnitool/gradio/agent/llm_utils"
    click EXE "https://github.com/shawnvirtly/omniparser/blob/master/omnitool/gradio/executor/anthropic_executor.py"
    click TSF "https://github.com/shawnvirtly/omniparser/tree/master/omnitool/gradio/tools"
    click LOOP "https://github.com/shawnvirtly/omniparser/blob/master/omnitool/gradio/loop.py"
    click BP "https://github.com/shawnvirtly/omniparser/blob/master/omnitool/omniparserserver/omniparserserver.py"
    click CPU "https://github.com/shawnvirtly/omniparser/tree/master/util"
    click VMControl "https://github.com/shawnvirtly/omniparser/tree/master/omnitool/omnibox"
    click VMBuild "https://github.com/shawnvirtly/omniparser/tree/master/omnitool/omnibox/vm"

    %% Styles
    classDef ui fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef agent fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef parser fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef vm fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef external fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
```
