# showlab/computer_use_ootb architecture

> Architecture diagram of the GitHub repository showlab/computer_use_ootb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/showlab/computer_use_ootb
- Repository: https://github.com/showlab/computer_use_ootb

## Overview

##### Project type and purpose
- This is a Python-based desktop GUI agent framework (“Computer Use OOTB”) that exposes a Gradio web interface for controlling a local or remote computer via natural-language commands. It combines:
- A front-end web interface (Gradio)
- A back-end orchestration layer (planner, actor, executor, loop)
- A toolkit for OS-level actions (screen capture, mouse/keyboard simulation, file edits)
- Integration with external LLM APIs (Anthropic Claude 3.5, OpenAI GPT-4o, Qwen)
- Optional local VLA models (ShowUI, UI-TARS)

##### File structure
- Top-level files:
- app.py: entry point that launches the Gradio interface
- computer_use_demo/: core logic
- install_tools/: scripts to install local models (ShowUI, UI-TARS)
- assets/: examples, logos, UI snapshots
- requirements.txt, README.md, .gitignore, LICENSE
- computer_use_demo/ contains:
- gui_agent/actor: low-level vision-language-action modules
- gui_agent/planner: high-level planning modules
- executor: modules that actually invoke the actor or API
- loop.py: main orchestration loop (intake user prompt → plan → act → feedback)
- remote_inference.py: remote planning support via SSH
- tools/: abstractions for OS operations (bash, screen capture, editing, logging)
- No persistent database; state flows through in-memory loop.

##### README for additional insights
- The README describes OOTB as a “full-stack” Desktop GUI agent supporting both API-based and local ML models.
- Highlights “Unified Model” vs separate Planner/Actor roles.
- Presents the user workflow:
user → Gradio UI → planner → actor → tools → OS actions → screenshot → planner …

- Blue rectangles for internal modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User & UI Layer
    subgraph "User & UI"
        direction TB
        Browser["Web/Mobile Browser"]:::external
        Gradio["Gradio Server\n(app.py)"]:::internal
        Assets["Assets & UI Examples"]:::internal
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        Loop["Main Loop\n(loop.py)"]:::internal
        RemoteInf["RemoteInference\n(SSH)"]:::optional
    end

    %% Planner Layer
    subgraph "Planner Modules"
        direction TB
        AnthroPlan["Anthropic API Planner"]:::internal
        APIVLMPlan["API-based VLM Planner"]:::internal
        LocalVLMPlan["Local VLM Planner"]:::internal
    end

    %% Actor & Executor Layer
    subgraph "Actor & Executor"
        direction TB
        ShowUIActor["ShowUI Actor"]:::internal
        UITARSActor["UI-TARS Actor"]:::internal
        AnthroExec["Anthropic Executor"]:::internal
        ShowUIExec["ShowUI Executor"]:::internal
    end

    %% LLM Utilities
    subgraph "LLM Utilities"
        direction TB
        LLMUtils["Core LLM Helpers"]:::internal
        OAI["OpenAI Client"]:::internal
        QWEN["Qwen Client"]:::internal
        RunLLM["LLM Runner Interface"]:::internal
    end

    %% Tools Library
    subgraph "Tools Library"
        direction TB
        BaseTool["Base Interface"]:::internal
        BashTool["Bash Interactions"]:::internal
        CollectTool["File Collection"]:::internal
        ColorLog["Colored Text Logging"]:::internal
        ComputerTool["Mouse/Keyboard Control"]:::internal
        EditTool["File Editing"]:::internal
        LoggerTool["Logging Utilities"]:::internal
        RunTool["Command Running"]:::internal
        ScreenCap["Screen Capture"]:::internal
    end

    %% External Services
    subgraph "External Services"
        direction TB
        AnthroAPI["Anthropic Claude 3.5 API"]:::external
        OpenAIAPI["OpenAI GPT-4o API"]:::external
        QwenAPI["Qwen API"]:::external
        UITARSServer["UI-TARS Server"]:::optional
    end

    %% Operating System
    subgraph "Operating System"
        direction TB
        ScreenBuf["Screen Buffer"]:::os
        WindowMgr["Window Manager"]:::os
        FileSys["File System"]:::os
    end

    %% Connections
    Browser -->|"HTTP Request"| Gradio
    Assets --> Gradio
    Gradio -->|"invoke"| Loop
    Loop -->|"plan"| AnthroPlan
    Loop -->|"plan"| APIVLMPlan
    Loop -->|"plan"| LocalVLMPlan
    AnthroPlan -->|"calls API"| AnthroAPI
    APIVLMPlan -->|"calls API"| OpenAIAPI
    APIVLMPlan -->|"calls API"| QwenAPI
    LocalVLMPlan -->|"runs locally"| UITARSServer
    AnthroPlan -->|">Planner output"| RunLLM
    APIVLMPlan -->|">Planner output"| RunLLM
    LocalVLMPlan -->|">Planner output"| RunLLM
    RunLLM -->|"output"| AnthroExec
    RunLLM -->|"output"| ShowUIExec
    ShowUIExec -->|"action commands"| BaseTool
    AnthroExec -->|"action commands"| BaseTool
    BaseTool --> BashTool
    BaseTool --> CollectTool
    BaseTool --> ColorLog
    BaseTool --> ComputerTool
    BaseTool --> EditTool
    BaseTool --> LoggerTool
    BaseTool --> RunTool
    BaseTool --> ScreenCap
    ScreenCap --> ScreenBuf
    ComputerTool --> WindowMgr
    EditTool --> FileSys
    Loop --> ScreenCap
    Loop --> ShowUIActor
    Loop --> UITARSActor
    ShowUIActor --> ShowUIExec
    UITARSActor --> ShowUIExec
    Loop --> RemoteInf
    RemoteInf --> Loop

    %% Click Events
    click Gradio "https://github.com/showlab/computer_use_ootb/blob/main/app.py"
    click Assets "https://github.com/showlab/computer_use_ootb/tree/main/assets/"
    click Loop "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/loop.py"
    click RemoteInf "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/remote_inference.py"
    click AnthroPlan "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/planner/anthropic_agent.py"
    click APIVLMPlan "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/planner/api_vlm_planner.py"
    click LocalVLMPlan "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/planner/local_vlm_planner.py"
    click ShowUIActor "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/actor/showui_agent.py"
    click UITARSActor "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/actor/uitars_agent.py"
    click AnthroExec "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/executor/anthropic_executor.py"
    click ShowUIExec "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/executor/showui_executor.py"
    click LLMUtils "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/llm_utils/llm_utils.py"
    click OAI "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/llm_utils/oai.py"
    click QWEN "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/llm_utils/qwen.py"
    click RunLLM "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/gui_agent/llm_utils/run_llm.py"
    click BaseTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/base.py"
    click BashTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/bash.py"
    click CollectTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/collection.py"
    click ColorLog "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/colorful_text.py"
    click ComputerTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/computer.py"
    click EditTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/edit.py"
    click LoggerTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/logger.py"
    click RunTool "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/run.py"
    click ScreenCap "https://github.com/showlab/computer_use_ootb/blob/main/computer_use_demo/tools/screen_capture.py"

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#1F77B4,color:#000
    classDef external fill:#DFFFE0,stroke:#2CA02C,color:#000
    classDef os fill:#FFE8D0,stroke:#FF7F0E,color:#000
    classDef optional stroke-dasharray: 5 5,fill:#F0F0F0,stroke:#7F7F7F,color:#000
```
