How xlang-ai/osworld works
OSWorld is a benchmark and desktop-control environment for evaluating computer-use agents. A run configures a DesktopEnv with a VM provider, prepares a desktop, accepts agent actions, and evaluates the resulting state with task-specific getters and metrics. The README documents VMware, VirtualBox, Docker, Modal, Daytona, and cloud-provider options; sampled source confirms provider abstractions and several agent/LLM subsystems. Most workflow links below are documented rather than verified from runtime entry-point source.
Main entry points: providers VM Providers · mm_agents GUI Agents · desktop_env.py DesktopEnv · conversable_agent.py Agent Conversations
Components 26
Desktop Environment
- DesktopEnv
desktop_env/desktop_env.py - Provider Contract
desktop_env/providers/base.py - VM Providers
desktop_env/providers - VMware Provider
desktop_env/providers/vmware/manager.py - VirtualBox Provider
desktop_env/providers/virtualbox/manager.py - AWS Provider
desktop_env/providers/aws/manager.py - Other Providers
desktop_env/providers - Desktop VM
- Desktop Actions
desktop_env/actions.py
Task Evaluation
- Task Getters
desktop_env/evaluators/getters - Task Metrics
desktop_env/evaluators/metrics
Agent Implementations
- GUI Agents
mm_agents - Anthropic Agent
mm_agents/anthropic/main.py - AutoGLM Agent
mm_agents/autoglm/main.py - Maestro Agent
mm_agents/maestro/core/engine.py - VLAA Agent
mm_agents/vlaa_gui/agents/worker.py
Agent Framework
- Agent Conversations
mm_agents/coact/autogen/agentchat/conversable_agent.py - Group Chat
mm_agents/coact/autogen/agentchat/groupchat.py - Agent Selection
mm_agents/coact/autogen/agentchat/group/patterns/random.py - Group Tool Executor
mm_agents/coact/autogen/agentchat/group/group_tool_executor.py - Handoff Targets
mm_agents/coact/autogen/agentchat/group/targets/transition_target.py - LLM Configuration
mm_agents/coact/autogen/llm_config.py - Anthropic Client
mm_agents/coact/autogen/oai/anthropic.py - Agent I/O
mm_agents/coact/autogen/io
Other components
- Benchmark User
- Run Interface
quickstart.py
Connections 27
- Benchmark User → Run Interface: starts run
README.md - Run Interface → DesktopEnv: configures
- DesktopEnv → Provider Contract: uses provider
- Provider Contract → VM Providers: implemented by
desktop_env/providers/base.py - VM Providers → VMware Provider: selects
README.md - VM Providers → VirtualBox Provider: selects
README.md - VM Providers → AWS Provider: selects
README.md - VM Providers → Other Providers: selects
README.md - VMware Provider → Desktop VM: hosts
README.md - VirtualBox Provider → Desktop VM: hosts
README.md - AWS Provider → Desktop VM: hosts
README.md - Other Providers → Desktop VM: hosts
README.md - DesktopEnv → Desktop Actions: executes
- GUI Agents → Desktop Actions: issues
- DesktopEnv → Task Getters: checks state
- Task Getters → Task Metrics: produces scores
- Anthropic Agent → GUI Agents: implements
mm_agents/anthropic/main.py - AutoGLM Agent → GUI Agents: implements
mm_agents/autoglm/main.py - Maestro Agent → GUI Agents: implements
mm_agents/maestro/core/engine.py - VLAA Agent → GUI Agents: implements
mm_agents/vlaa_gui/agents/worker.py - Agent Conversations → Group Chat: coordinates
- Group Chat → Agent Selection: selects agents
mm_agents/coact/autogen/agentchat/group/patterns/random.py - Agent Selection → Handoff Targets: sets handoffs
mm_agents/coact/autogen/agentchat/group/patterns/random.py - Group Chat → Group Tool Executor: routes tools
- Agent Conversations → LLM Configuration: uses config
- LLM Configuration → Anthropic Client: configures
mm_agents/coact/autogen/oai/anthropic.py - Agent Conversations → Agent I/O: communicates via