# openai/openai-cua-sample-app architecture

> Architecture diagram of the GitHub repository openai/openai-cua-sample-app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/openai-cua-sample-app
- Repository: https://github.com/openai/openai-cua-sample-app

## Overview

##### Project Type & Purpose
- This is a sample application for “Computer Using Agent (CUA),” which leverages the OpenAI API to interact with computer environments. It demonstrates how an agent can process UI screenshots and execute actions (like clicking, typing, etc.) on different computer environments.
- The README makes it clear that it uses two primary abstractions (an Agent abstraction and a Computer abstraction) to encapsulate the logic for turning model outputs into computer actions.

##### File Structure
- Key directories and files:
- Top-level files: .env.example, Dockerfile, README.md, etc. These indicate that the project supports both local and containerized deployment.
- The agent/ folder (with agent/agent.py) which contains the agent logic.
- The computers/ folder holds various implementations (or “computer environments”) such as local playwright, Docker, Browserbase, and Scrapybara which are different strategies for executing actions.
- The CLI (cli.py) is the entry point that accepts command-line arguments and chooses the computer environment to use.
- examples/ and simple_cua_loop.py provide usage samples and demonstrate how to run the loop.
- Notice that the project architecture separates concerns: one abstraction for the dialogue loop/agent and another for executing computer interactions. This implies a modular design that communicates via defined interfaces.

##### Extract Insights from the README
- The README outlines the communications flow:
- The Agent sends computer actions (e.g., click, type) based on screenshots or model responses.
- The computer modules strictly execute those actions in their respective environments.
- There is detailed documentation on the supported “computer actions” and different implementation environments.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Core Components
    CLI["CLI"]:::internal
    AgentLoop["Agent Loop"]:::internal
    OpenAIAPI["OpenAI API"]:::external

    CLI -->|"start"| AgentLoop
    AgentLoop -->|"ModelCall"| OpenAIAPI
    OpenAIAPI -->|"ModelResponse"| AgentLoop

    %% Computer Abstraction and Implementations
    subgraph "Computer Abstraction & Implementations"
        ComputerInterface["Computer Interface"]:::internal
        LocalPlaywright["Local Playwright"]:::internal
        DockerImpl["Docker Implementation"]:::internal
        BrowserbaseImpl["Browserbase Implementation"]:::internal
        ScrapybaraImpl["Scrapybara Implementation"]:::internal
    end

    AgentLoop -->|"ActionCommand"| ComputerInterface
    ComputerInterface -->|"ExecuteAction"| AgentLoop

    ComputerInterface -->|"impl"| LocalPlaywright
    ComputerInterface -->|"impl"| DockerImpl
    ComputerInterface -->|"impl"| BrowserbaseImpl
    ComputerInterface -->|"impl"| ScrapybaraImpl

    %% Examples & Utility Scripts
    subgraph "Examples & Utilities"
        Example1["Function Calling Example"]:::example
        Example2["Playwright with Custom Functions"]:::example
        Example3["Weather Example"]:::example
        SimpleCUALoop["Simple CUA Loop"]:::example
    end

    CLI -->|"demoUsage"| Example1
    CLI -->|"demoUsage"| Example2
    CLI -->|"demoUsage"| Example3
    CLI -->|"demoUsage"| SimpleCUALoop

    %% Click Events
    click CLI "https://github.com/openai/openai-cua-sample-app/blob/main/cli.py"
    click AgentLoop "https://github.com/openai/openai-cua-sample-app/blob/main/agent/agent.py"
    click ComputerInterface "https://github.com/openai/openai-cua-sample-app/blob/main/computers/computer.py"
    click LocalPlaywright "https://github.com/openai/openai-cua-sample-app/blob/main/computers/local_playwright.py"
    click DockerImpl "https://github.com/openai/openai-cua-sample-app/blob/main/computers/docker.py"
    click BrowserbaseImpl "https://github.com/openai/openai-cua-sample-app/blob/main/computers/browserbase.py"
    click ScrapybaraImpl "https://github.com/openai/openai-cua-sample-app/blob/main/computers/scrapybara.py"
    click Example1 "https://github.com/openai/openai-cua-sample-app/blob/main/examples/function_calling_example.py"
    click Example2 "https://github.com/openai/openai-cua-sample-app/blob/main/examples/playwright_with_custom_functions.py"
    click Example3 "https://github.com/openai/openai-cua-sample-app/blob/main/examples/weather_example.py"
    click SimpleCUALoop "https://github.com/openai/openai-cua-sample-app/blob/main/simple_cua_loop.py"

    %% Styles
    classDef internal fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef external fill:#f5c6cb,stroke:#721c24,stroke-width:2px;
    classDef example fill:#d4edda,stroke:#155724,stroke-width:2px;
```
