# microsoft/omniparser architecture

> Architecture diagram of the GitHub repository microsoft/omniparser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/omniparser
- Repository: https://github.com/microsoft/omniparser

## Overview

OmniParser turns screenshots into structured UI elements; OmniTool uses that output in a computer-use agent workflow that selects and executes actions against a Windows VM. The main runtime path is UI → sampling loop → screenshot/parser client → parsing service, with an agent choosing actions and an executor applying tool calls. The repository also supports multiple model providers and orchestrated agents. README documents VM control and model choices; some VM and UI wiring is unsampled, so those relationships are omitted or marked unverified.

Main entry points: [`omnitool/gradio/loop.py`](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/loop.py) (Sampling loop), [`omnitool/gradio/agent/vlm_agent_with_orchestrator.py`](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent_with_orchestrator.py) (Orchestrated agent), [`omnitool/gradio/agent/llm_utils/omniparserclient.py`](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/omniparserclient.py) (Parser client), [`util/utils.py`](https://github.com/microsoft/omniparser/blob/master/util/utils.py) (Vision processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_app["Gradio app<br/>[app_new.py]"]
end

subgraph group_parsing["Screen parsing"]
  node_parserclient["Parser client"]
  node_parseapi["Parsing service"]
  node_parser["Screenshot parser<br/>[omniparser.py]"]
  node_visionutils["Vision processing<br/>[utils.py]"]
  node_detector["YOLO detector<br/>[yolov9.py]"]
  node_annotator["Box annotation<br/>[box_annotator.py]"]
end

subgraph group_agents["Agent runtime"]
  node_loop["Sampling loop<br/>[loop.py]"]
  node_vlm["Vision agent<br/>[vlm_agent.py]"]
  node_orchestrator["Orchestrated agent"]
  node_anthropic["Anthropic agent<br/>[anthropic_agent.py]"]
  node_openai["OpenAI client<br/>[oaiclient.py]"]
  node_groq["Groq client<br/>[groqclient.py]"]
  node_trajectory["Trajectory log"]
end

subgraph group_execution["Computer execution"]
  node_capture["Screenshot capture<br/>[screen_capture.py]"]
  node_executor["Tool executor"]
  node_computer["Computer tool<br/>[computer.py]"]
  node_windows["Windows VM"]
end

node_user(("User"))

node_user -->|"uses"| node_app
node_app -->|"starts"| node_loop
node_loop -->|"requests parsing"| node_parserclient
node_parserclient -->|"captures screenshot"| node_capture
node_parserclient -.->|"posts screenshot"| node_parseapi
node_parseapi -->|"invokes parser"| node_parser
node_parser -->|"uses vision routines"| node_visionutils
node_visionutils -->|"loads detector"| node_detector
node_visionutils -->|"annotates detections"| node_annotator
node_loop -->|"selects agent"| node_vlm
node_loop -->|"selects agent"| node_orchestrator
node_loop -->|"selects agent"| node_anthropic
node_orchestrator -->|"calls model"| node_openai
node_orchestrator -->|"calls model"| node_groq
node_vlm -->|"uses client"| node_openai
node_vlm -->|"uses client"| node_groq
node_loop -->|"executes tool calls"| node_executor
node_executor -->|"dispatches tools"| node_computer
node_computer -->|"uses screenshots"| node_capture
node_computer -.->|"controls VM"| node_windows
node_orchestrator -->|"writes steps"| node_trajectory

click node_app "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/app_new.py"
click node_loop "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/loop.py"
click node_capture "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/tools/screen_capture.py"
click node_parserclient "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/omniparserclient.py"
click node_parseapi "https://github.com/microsoft/omniparser/blob/master/omnitool/omniparserserver/omniparserserver.py"
click node_parser "https://github.com/microsoft/omniparser/blob/master/util/omniparser.py"
click node_visionutils "https://github.com/microsoft/omniparser/blob/master/util/utils.py"
click node_detector "https://github.com/microsoft/omniparser/blob/master/util/yolov9.py"
click node_annotator "https://github.com/microsoft/omniparser/blob/master/util/box_annotator.py"
click node_vlm "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent.py"
click node_orchestrator "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent_with_orchestrator.py"
click node_anthropic "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/anthropic_agent.py"
click node_openai "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/oaiclient.py"
click node_groq "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/groqclient.py"
click node_executor "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/executor/anthropic_executor.py"
click node_computer "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/tools/computer.py"
click node_windows "https://github.com/microsoft/omniparser/tree/master/omnitool/omnibox"
click node_trajectory "https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent_with_orchestrator.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_user toneBlue
class node_parserclient,node_parseapi,node_parser,node_visionutils,node_detector,node_annotator toneAmber
class node_loop,node_vlm,node_orchestrator,node_anthropic,node_openai,node_groq,node_trajectory toneMint
class node_capture,node_executor,node_computer,node_windows toneRose
```

## Components

### User interface

- [Gradio app](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/app_new.py): `omnitool/gradio/app_new.py`

### Screen parsing

- [Parser client](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/omniparserclient.py): `omnitool/gradio/agent/llm_utils/omniparserclient.py`
- [Parsing service](https://github.com/microsoft/omniparser/blob/master/omnitool/omniparserserver/omniparserserver.py): `omnitool/omniparserserver/omniparserserver.py`
- [Screenshot parser](https://github.com/microsoft/omniparser/blob/master/util/omniparser.py): `util/omniparser.py`
- [Vision processing](https://github.com/microsoft/omniparser/blob/master/util/utils.py): `util/utils.py`
- [YOLO detector](https://github.com/microsoft/omniparser/blob/master/util/yolov9.py): `util/yolov9.py`
- [Box annotation](https://github.com/microsoft/omniparser/blob/master/util/box_annotator.py): `util/box_annotator.py`

### Agent runtime

- [Sampling loop](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/loop.py): `omnitool/gradio/loop.py`
- [Vision agent](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent.py): `omnitool/gradio/agent/vlm_agent.py`
- [Orchestrated agent](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent_with_orchestrator.py): `omnitool/gradio/agent/vlm_agent_with_orchestrator.py`
- [Anthropic agent](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/anthropic_agent.py): `omnitool/gradio/agent/anthropic_agent.py`
- [OpenAI client](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/oaiclient.py): `omnitool/gradio/agent/llm_utils/oaiclient.py`
- [Groq client](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/llm_utils/groqclient.py): `omnitool/gradio/agent/llm_utils/groqclient.py`
- [Trajectory log](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/agent/vlm_agent_with_orchestrator.py): `omnitool/gradio/agent/vlm_agent_with_orchestrator.py`

### Computer execution

- [Screenshot capture](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/tools/screen_capture.py): `omnitool/gradio/tools/screen_capture.py`
- [Tool executor](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/executor/anthropic_executor.py): `omnitool/gradio/executor/anthropic_executor.py`
- [Computer tool](https://github.com/microsoft/omniparser/blob/master/omnitool/gradio/tools/computer.py): `omnitool/gradio/tools/computer.py`
- [Windows VM](https://github.com/microsoft/omniparser/tree/master/omnitool/omnibox): `omnitool/omnibox`

### Other components

- **User**

## Connections

- User → Gradio app: uses
- Gradio app → Sampling loop: starts (evidence: `omnitool/gradio/app_new.py`)
- Sampling loop → Parser client: requests parsing (evidence: `omnitool/gradio/loop.py`)
- Parser client → Screenshot capture: captures screenshot (evidence: `omnitool/gradio/agent/llm_utils/omniparserclient.py`)
- Parser client → Parsing service: posts screenshot
- Parsing service → Screenshot parser: invokes parser (evidence: `omnitool/omniparserserver/omniparserserver.py`)
- Screenshot parser → Vision processing: uses vision routines (evidence: `util/omniparser.py`)
- Vision processing → YOLO detector: loads detector (evidence: `util/utils.py`)
- Vision processing → Box annotation: annotates detections (evidence: `util/utils.py`)
- Sampling loop → Vision agent: selects agent (evidence: `omnitool/gradio/loop.py`)
- Sampling loop → Orchestrated agent: selects agent (evidence: `omnitool/gradio/loop.py`)
- Sampling loop → Anthropic agent: selects agent (evidence: `omnitool/gradio/loop.py`)
- Orchestrated agent → OpenAI client: calls model (evidence: `omnitool/gradio/agent/vlm_agent_with_orchestrator.py`)
- Orchestrated agent → Groq client: calls model (evidence: `omnitool/gradio/agent/vlm_agent_with_orchestrator.py`)
- Vision agent → OpenAI client: uses client (evidence: `omnitool/gradio/agent/vlm_agent.py`)
- Vision agent → Groq client: uses client (evidence: `omnitool/gradio/agent/vlm_agent.py`)
- Sampling loop → Tool executor: executes tool calls (evidence: `omnitool/gradio/loop.py`)
- Tool executor → Computer tool: dispatches tools (evidence: `omnitool/gradio/executor/anthropic_executor.py`)
- Computer tool → Screenshot capture: uses screenshots (evidence: `omnitool/gradio/tools/computer.py`)
- Computer tool → Windows VM: controls VM
- Orchestrated agent → Trajectory log: writes steps (evidence: `omnitool/gradio/agent/vlm_agent_with_orchestrator.py`)
