# e2b-dev/open-computer-use architecture

> Architecture diagram of the GitHub repository e2b-dev/open-computer-use, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/e2b-dev/open-computer-use
- Repository: https://github.com/e2b-dev/open-computer-use

## Overview

##### Project Type and Purpose
- The project is a secure cloud-based Linux computer system that leverages the E2B Desktop Sandbox. It is controlled by open-source LLMs, meaning that it comprises components that interface with various language model providers (e.g., Groq, OpenAI, Hugging Face) and manages interactions through configurable modules.
- The design is primarily focused on providing a secure sandboxed desktop environment that the user can control through keyboard, mouse, shell commands, and also view via live streaming. The system also supports dynamic configuration of LLMs and integration with external APIs.

##### File Structure
- At the top level, key files include “main.py”, which likely serves as the entry point into the program, and “pyproject.toml” for dependency management.
- The directory “os_computer_use” contains most of the core modules, such as:
- browser_script.py: Likely handles interactions coming from or output to a browser interface.
- config.py: Contains configuration and setup for LLM providers.
- grounding.py, llm_provider.py, providers.py: Modules for the integration with LLMs. This establishes abstraction over multiple LLM providers.
- logging.py: Responsible for logging system events.
- sandbox.py and sandbox_agent.py: Handle the secure sandbox environment and the agent operations within that environment.
- streaming.py: Likely deals with live display streaming of the desktop environment.
- templates directory with log.html: Provides front-end templates possibly for logging or UI rendering.
- The “tests” directory includes unit tests for critical modules like “llm_provider.py” and “sandbox_agent.py”.

##### Key Details from the README
- The README clearly explains the feature set—secure desktop sandbox, integration with multiple LLM providers, streaming display, and control via keyboard/mouse/shell commands.
- It outlines how to swap in/out providers by modifying config.py and providers.py, emphasizing the modularity and extensibility of the LLM integration.
- Additional external documentation is referenced, making the project design clear: a separation between core sandbox functionality and LLM-based decision making.

a. Main Components:
- Sandbox Environment:
- Modules: sandbox.py, sandbox_agent.py, streaming.py.
- Purpose: Managing the isolated cloud Linux instance, live display streaming, and agent control.
- LLM Integration Layer:
- Modules: llm_provider.py, grounding.py, providers.py, config.py.
- Purpose: Interfacing with various LLM providers, including support for vision and action tasks.
- User Interface Component:
- Components: browser_script.py and the web UI templates (log.html in “templates”).
- Purpose: Allows users to interact with the system, observe streamed output, and provide input.
- Core Controller/Entry Point:
- Module: main.py.
- Purpose: Boots up the application, initializes the environment, and orchestrates component interactions.
- Supporting Components:
- logging.py for system logging.
- Additional directories for assets (including pre-made architecture images) and configuration files (like pyproject.toml).

b. Relationships and Interactions:
- Main.py initializes and coordinates interactions between the sandbox, LLM integration, and the UI.
- The sandbox (sandbox.py/sandbox_agent.py) communicates with the streaming module to provide live display data to the user interface.
- The LLM integration layer (providers, config, and llm_provider modules) processes user inputs and system commands, routing them via configured third-party LLM APIs.
- The browser_script acts as a client side component that likely sends user commands and displays streamed results.
- Logging is captured and may be rendered using templates (log.html) and reported throughout the system.

c. Important Architectural Patterns and Design Principles:
- Modular Design: Clear separation of concerns between sandbox operations, LLM provider configurations, and UI interactions.
- Extensibility: The use of a providers module (and associated config) permits easy integration of new LLM providers.
- Secure Operation: Emphasis on sandboxing a cloud-based Linux environment ensures operational security.
- Event Flow and Streaming: A data flow from user actions (via browser script/UI) to LLM processing then resulting in streamed feedback.

d. Relevant Technologies, Frameworks, or Libraries:
- Python is the core language with version 3.10+ required.
- E2B Desktop Sandbox is integrated for secure desktop isolation.
- The system integrates multiple third-party LLM APIs (OpenAI, Groq, Anthropic, etc.).
- The use of Poetry for dependency management and environment configuration (pyproject.toml) is noted.
- Frontend web technologies may be implied via browser_script.py and HTML templates.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface"
        UI_BS["Browser Script (UI)"]:::ui
        UI_LOG["Log Template (UI)"]:::ui
    end

    %% Core Controller / Entry Point
    subgraph "Core Controller"
        CTL["Core Controller (main.py)"]:::backend
    end

    %% Backend Processing Layer
    subgraph "Backend Processing"
        %% Sandbox Environment Subgraph
        subgraph "Sandbox Environment"
            S1["Sandbox Module (sandbox.py)"]:::sandbox
            S2["Sandbox Agent (sandbox_agent.py)"]:::sandbox
            S3["Live Streaming (streaming.py)"]:::sandbox
        end

        %% LLM Integration Subgraph
        subgraph "LLM Integration Layer"
            L1["LLM Provider (llm_provider.py)"]:::llm
            L2["Grounding (grounding.py)"]:::llm
            L3["Providers (providers.py)"]:::llm
            L4["Configuration (config.py)"]:::llm
        end
    end

    %% Logging Service
    LOG["Logging Service (logging.py)"]:::logging

    %% Interactions
    UI_BS -->|"UserInput"| CTL
    CTL -->|"InitSandbox"| S1
    CTL -->|"InitSandbox"| S2
    CTL -->|"InitSandbox"| S3
    CTL -->|"ProcessCommand"| L1
    CTL -->|"ProcessCommand"| L2
    CTL -->|"ProcessCommand"| L3
    CTL -->|"ProcessCommand"| L4
    S3 -->|"StreamsData"| UI_BS

    %% Logging arrows from all components to Logging Service
    CTL -->|"LogEvent"| LOG
    S1 -->|"LogEvent"| LOG
    S2 -->|"LogEvent"| LOG
    S3 -->|"LogEvent"| LOG
    L1 -->|"LogEvent"| LOG
    L2 -->|"LogEvent"| LOG
    L3 -->|"LogEvent"| LOG
    L4 -->|"LogEvent"| LOG
    UI_BS -->|"LogEvent"| LOG
    UI_LOG -->|"LogEvent"| LOG

    %% Click Events
    click S1 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/sandbox.py"
    click S2 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/sandbox_agent.py"
    click S3 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/streaming.py"
    click L1 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/llm_provider.py"
    click L2 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/grounding.py"
    click L3 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/providers.py"
    click L4 "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/config.py"
    click UI_BS "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/browser_script.py"
    click UI_LOG "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/templates/log.html"
    click CTL "https://github.com/e2b-dev/open-computer-use/blob/master/main.py"
    click LOG "https://github.com/e2b-dev/open-computer-use/blob/master/os_computer_use/logging.py"

    %% Styles
    classDef ui fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef backend fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef sandbox fill:#D5F5E3,stroke:#333,stroke-width:2px;
    classDef llm fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef logging fill:#E8DAEF,stroke:#333,stroke-width:2px;
```
