How microsoft/omniparser works
OmniParser turns screenshots into structured UI elements; OmniTool uses that output in a computer-use agent workflow that selects and executes actions against a Windows VM. The main runtime path is UI → sampling loop → screenshot/parser client → parsing service, with an agent choosing actions and an executor applying tool calls. The repository also supports multiple model providers and orchestrated agents. README documents VM control and model choices; some VM and UI wiring is unsampled, so those relationships are omitted or marked unverified.
Main entry points: loop.py Sampling loop · vlm_agent_with_orchestrator.py Orchestrated agent · omniparserclient.py Parser client · utils.py Vision processing
Components 19
User interface
- Gradio app
omnitool/gradio/app_new.py
Screen parsing
- Parser client
omnitool/gradio/agent/llm_utils/omniparserclient.py - Parsing service
omnitool/omniparserserver/omniparserserver.py - Screenshot parser
util/omniparser.py - Vision processing
util/utils.py - YOLO detector
util/yolov9.py - Box annotation
util/box_annotator.py
Agent runtime
- Sampling loop
omnitool/gradio/loop.py - Vision agent
omnitool/gradio/agent/vlm_agent.py - Orchestrated agent
omnitool/gradio/agent/vlm_agent_with_orchestrator.py - Anthropic agent
omnitool/gradio/agent/anthropic_agent.py - OpenAI client
omnitool/gradio/agent/llm_utils/oaiclient.py - Groq client
omnitool/gradio/agent/llm_utils/groqclient.py - Trajectory log
omnitool/gradio/agent/vlm_agent_with_orchestrator.py
Computer execution
- Screenshot capture
omnitool/gradio/tools/screen_capture.py - Tool executor
omnitool/gradio/executor/anthropic_executor.py - Computer tool
omnitool/gradio/tools/computer.py - Windows VM
omnitool/omnibox
Other components
- User
Connections 21
- User → Gradio app: uses
- Gradio app → Sampling loop: starts
omnitool/gradio/app_new.py - Sampling loop → Parser client: requests parsing
omnitool/gradio/loop.py - Parser client → Screenshot capture: captures screenshot
omnitool/gradio/agent/llm_utils/omniparserclient.py - Parser client → Parsing service: posts screenshot
- Parsing service → Screenshot parser: invokes parser
omnitool/omniparserserver/omniparserserver.py - Screenshot parser → Vision processing: uses vision routines
util/omniparser.py - Vision processing → YOLO detector: loads detector
util/utils.py - Vision processing → Box annotation: annotates detections
util/utils.py - Sampling loop → Vision agent: selects agent
omnitool/gradio/loop.py - Sampling loop → Orchestrated agent: selects agent
omnitool/gradio/loop.py - Sampling loop → Anthropic agent: selects agent
omnitool/gradio/loop.py - Orchestrated agent → OpenAI client: calls model
omnitool/gradio/agent/vlm_agent_with_orchestrator.py - Orchestrated agent → Groq client: calls model
omnitool/gradio/agent/vlm_agent_with_orchestrator.py - Vision agent → OpenAI client: uses client
omnitool/gradio/agent/vlm_agent.py - Vision agent → Groq client: uses client
omnitool/gradio/agent/vlm_agent.py - Sampling loop → Tool executor: executes tool calls
omnitool/gradio/loop.py - Tool executor → Computer tool: dispatches tools
omnitool/gradio/executor/anthropic_executor.py - Computer tool → Screenshot capture: uses screenshots
omnitool/gradio/tools/computer.py - Computer tool → Windows VM: controls VM
- Orchestrated agent → Trajectory log: writes steps
omnitool/gradio/agent/vlm_agent_with_orchestrator.py