# xlang-ai/osworld architecture

> Architecture diagram of the GitHub repository xlang-ai/osworld, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xlang-ai/osworld
- Repository: https://github.com/xlang-ai/osworld

## Overview

OSWorld is a benchmark and desktop-control environment for evaluating computer-use agents. A run configures a DesktopEnv with a VM provider, prepares a desktop, accepts agent actions, and evaluates the resulting state with task-specific getters and metrics. The README documents VMware, VirtualBox, Docker, Modal, Daytona, and cloud-provider options; sampled source confirms provider abstractions and several agent/LLM subsystems. Most workflow links below are documented rather than verified from runtime entry-point source.

Main entry points: [`desktop_env/providers`](https://github.com/xlang-ai/osworld/tree/main/desktop_env/providers) (VM Providers), [`mm_agents`](https://github.com/xlang-ai/osworld/tree/main/mm_agents) (GUI Agents), [`desktop_env/desktop_env.py`](https://github.com/xlang-ai/osworld/blob/main/desktop_env/desktop_env.py) (DesktopEnv), [`mm_agents/coact/autogen/agentchat/conversable_agent.py`](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/conversable_agent.py) (Agent Conversations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_environment["Desktop Environment"]
  node_desktop_env["DesktopEnv<br/>[desktop_env.py]"]
  node_provider_base["Provider Contract<br/>[base.py]"]
  node_provider_managers["VM Providers"]
  node_vmware["VMware Provider<br/>[manager.py]"]
  node_virtualbox["VirtualBox Provider<br/>[manager.py]"]
  node_aws["AWS Provider<br/>[manager.py]"]
  node_other_providers["Other Providers"]
  node_desktop_vm["Desktop VM"]
  node_actions["Desktop Actions<br/>[actions.py]"]
end

subgraph group_evaluation["Task Evaluation"]
  node_evaluators["Task Getters"]
  node_metrics["Task Metrics"]
end

subgraph group_agents["Agent Implementations"]
  node_agents["GUI Agents"]
  node_anthropic_agent["Anthropic Agent<br/>[main.py]"]
  node_autoglm["AutoGLM Agent<br/>[main.py]"]
  node_maestro["Maestro Agent<br/>[engine.py]"]
  node_vlaa["VLAA Agent<br/>[worker.py]"]
end

subgraph group_agent_framework["Agent Framework"]
  node_autogen_chat["Agent Conversations"]
  node_group_chat["Group Chat<br/>[groupchat.py]"]
  node_group_patterns["Agent Selection<br/>[random.py]"]
  node_tool_executor["Group Tool Executor"]
  node_transition_target["Handoff Targets"]
  node_llm_config["LLM Configuration<br/>[llm_config.py]"]
  node_anthropic_client["Anthropic Client<br/>[anthropic.py]"]
  node_io["Agent I/O"]
end

node_user(("Benchmark User"))
node_quickstart["Run Interface<br/>[quickstart.py]"]

node_user -->|"starts run"| node_quickstart
node_quickstart -.->|"configures"| node_desktop_env
node_desktop_env -.->|"uses provider"| node_provider_base
node_provider_base -->|"implemented by"| node_provider_managers
node_provider_managers -->|"selects"| node_vmware
node_provider_managers -->|"selects"| node_virtualbox
node_provider_managers -->|"selects"| node_aws
node_provider_managers -->|"selects"| node_other_providers
node_vmware -->|"hosts"| node_desktop_vm
node_virtualbox -->|"hosts"| node_desktop_vm
node_aws -->|"hosts"| node_desktop_vm
node_other_providers -->|"hosts"| node_desktop_vm
node_desktop_env -.->|"executes"| node_actions
node_agents -.->|"issues"| node_actions
node_desktop_env -.->|"checks state"| node_evaluators
node_evaluators -.->|"produces scores"| node_metrics
node_anthropic_agent -.->|"implements"| node_agents
node_autoglm -.->|"implements"| node_agents
node_maestro -.->|"implements"| node_agents
node_vlaa -.->|"implements"| node_agents
node_autogen_chat -.->|"coordinates"| node_group_chat
node_group_chat -->|"selects agents"| node_group_patterns
node_group_patterns -->|"sets handoffs"| node_transition_target
node_group_chat -.->|"routes tools"| node_tool_executor
node_autogen_chat -.->|"uses config"| node_llm_config
node_llm_config -->|"configures"| node_anthropic_client
node_autogen_chat -.->|"communicates via"| node_io

click node_quickstart "https://github.com/xlang-ai/osworld/blob/main/quickstart.py"
click node_desktop_env "https://github.com/xlang-ai/osworld/blob/main/desktop_env/desktop_env.py"
click node_provider_base "https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/base.py"
click node_provider_managers "https://github.com/xlang-ai/osworld/tree/main/desktop_env/providers"
click node_vmware "https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/vmware/manager.py"
click node_virtualbox "https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/virtualbox/manager.py"
click node_aws "https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/aws/manager.py"
click node_other_providers "https://github.com/xlang-ai/osworld/tree/main/desktop_env/providers"
click node_actions "https://github.com/xlang-ai/osworld/blob/main/desktop_env/actions.py"
click node_evaluators "https://github.com/xlang-ai/osworld/tree/main/desktop_env/evaluators/getters"
click node_metrics "https://github.com/xlang-ai/osworld/tree/main/desktop_env/evaluators/metrics"
click node_agents "https://github.com/xlang-ai/osworld/tree/main/mm_agents"
click node_anthropic_agent "https://github.com/xlang-ai/osworld/blob/main/mm_agents/anthropic/main.py"
click node_autoglm "https://github.com/xlang-ai/osworld/blob/main/mm_agents/autoglm/main.py"
click node_maestro "https://github.com/xlang-ai/osworld/blob/main/mm_agents/maestro/core/engine.py"
click node_vlaa "https://github.com/xlang-ai/osworld/blob/main/mm_agents/vlaa_gui/agents/worker.py"
click node_autogen_chat "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/conversable_agent.py"
click node_group_chat "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/groupchat.py"
click node_group_patterns "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/group/patterns/random.py"
click node_tool_executor "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/group/group_tool_executor.py"
click node_transition_target "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/group/targets/transition_target.py"
click node_llm_config "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/llm_config.py"
click node_anthropic_client "https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/oai/anthropic.py"
click node_io "https://github.com/xlang-ai/osworld/tree/main/mm_agents/coact/autogen/io"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_desktop_env,node_provider_base,node_provider_managers,node_vmware,node_virtualbox,node_aws,node_other_providers,node_desktop_vm,node_actions,node_user toneBlue
class node_evaluators,node_metrics toneAmber
class node_agents,node_anthropic_agent,node_autoglm,node_maestro,node_vlaa toneMint
class node_autogen_chat,node_group_chat,node_group_patterns,node_tool_executor,node_transition_target,node_llm_config,node_anthropic_client,node_io toneRose
class node_quickstart toneTeal
```

## Components

### Desktop Environment

- [DesktopEnv](https://github.com/xlang-ai/osworld/blob/main/desktop_env/desktop_env.py): `desktop_env/desktop_env.py`
- [Provider Contract](https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/base.py): `desktop_env/providers/base.py`
- [VM Providers](https://github.com/xlang-ai/osworld/tree/main/desktop_env/providers): `desktop_env/providers`
- [VMware Provider](https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/vmware/manager.py): `desktop_env/providers/vmware/manager.py`
- [VirtualBox Provider](https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/virtualbox/manager.py): `desktop_env/providers/virtualbox/manager.py`
- [AWS Provider](https://github.com/xlang-ai/osworld/blob/main/desktop_env/providers/aws/manager.py): `desktop_env/providers/aws/manager.py`
- [Other Providers](https://github.com/xlang-ai/osworld/tree/main/desktop_env/providers): `desktop_env/providers`
- **Desktop VM**
- [Desktop Actions](https://github.com/xlang-ai/osworld/blob/main/desktop_env/actions.py): `desktop_env/actions.py`

### Task Evaluation

- [Task Getters](https://github.com/xlang-ai/osworld/tree/main/desktop_env/evaluators/getters): `desktop_env/evaluators/getters`
- [Task Metrics](https://github.com/xlang-ai/osworld/tree/main/desktop_env/evaluators/metrics): `desktop_env/evaluators/metrics`

### Agent Implementations

- [GUI Agents](https://github.com/xlang-ai/osworld/tree/main/mm_agents): `mm_agents`
- [Anthropic Agent](https://github.com/xlang-ai/osworld/blob/main/mm_agents/anthropic/main.py): `mm_agents/anthropic/main.py`
- [AutoGLM Agent](https://github.com/xlang-ai/osworld/blob/main/mm_agents/autoglm/main.py): `mm_agents/autoglm/main.py`
- [Maestro Agent](https://github.com/xlang-ai/osworld/blob/main/mm_agents/maestro/core/engine.py): `mm_agents/maestro/core/engine.py`
- [VLAA Agent](https://github.com/xlang-ai/osworld/blob/main/mm_agents/vlaa_gui/agents/worker.py): `mm_agents/vlaa_gui/agents/worker.py`

### Agent Framework

- [Agent Conversations](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/conversable_agent.py): `mm_agents/coact/autogen/agentchat/conversable_agent.py`
- [Group Chat](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/groupchat.py): `mm_agents/coact/autogen/agentchat/groupchat.py`
- [Agent Selection](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/group/patterns/random.py): `mm_agents/coact/autogen/agentchat/group/patterns/random.py`
- [Group Tool Executor](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/group/group_tool_executor.py): `mm_agents/coact/autogen/agentchat/group/group_tool_executor.py`
- [Handoff Targets](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/agentchat/group/targets/transition_target.py): `mm_agents/coact/autogen/agentchat/group/targets/transition_target.py`
- [LLM Configuration](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/llm_config.py): `mm_agents/coact/autogen/llm_config.py`
- [Anthropic Client](https://github.com/xlang-ai/osworld/blob/main/mm_agents/coact/autogen/oai/anthropic.py): `mm_agents/coact/autogen/oai/anthropic.py`
- [Agent I/O](https://github.com/xlang-ai/osworld/tree/main/mm_agents/coact/autogen/io): `mm_agents/coact/autogen/io`

### Other components

- **Benchmark User**
- [Run Interface](https://github.com/xlang-ai/osworld/blob/main/quickstart.py): `quickstart.py`

## Connections

- Benchmark User → Run Interface: starts run (evidence: `README.md`)
- Run Interface → DesktopEnv: configures
- DesktopEnv → Provider Contract: uses provider
- Provider Contract → VM Providers: implemented by (evidence: `desktop_env/providers/base.py`)
- VM Providers → VMware Provider: selects (evidence: `README.md`)
- VM Providers → VirtualBox Provider: selects (evidence: `README.md`)
- VM Providers → AWS Provider: selects (evidence: `README.md`)
- VM Providers → Other Providers: selects (evidence: `README.md`)
- VMware Provider → Desktop VM: hosts (evidence: `README.md`)
- VirtualBox Provider → Desktop VM: hosts (evidence: `README.md`)
- AWS Provider → Desktop VM: hosts (evidence: `README.md`)
- Other Providers → Desktop VM: hosts (evidence: `README.md`)
- DesktopEnv → Desktop Actions: executes
- GUI Agents → Desktop Actions: issues
- DesktopEnv → Task Getters: checks state
- Task Getters → Task Metrics: produces scores
- Anthropic Agent → GUI Agents: implements (evidence: `mm_agents/anthropic/main.py`)
- AutoGLM Agent → GUI Agents: implements (evidence: `mm_agents/autoglm/main.py`)
- Maestro Agent → GUI Agents: implements (evidence: `mm_agents/maestro/core/engine.py`)
- VLAA Agent → GUI Agents: implements (evidence: `mm_agents/vlaa_gui/agents/worker.py`)
- Agent Conversations → Group Chat: coordinates
- Group Chat → Agent Selection: selects agents (evidence: `mm_agents/coact/autogen/agentchat/group/patterns/random.py`)
- Agent Selection → Handoff Targets: sets handoffs (evidence: `mm_agents/coact/autogen/agentchat/group/patterns/random.py`)
- Group Chat → Group Tool Executor: routes tools
- Agent Conversations → LLM Configuration: uses config
- LLM Configuration → Anthropic Client: configures (evidence: `mm_agents/coact/autogen/oai/anthropic.py`)
- Agent Conversations → Agent I/O: communicates via
