How e2b-dev/open-computer-use works
Project Type and Purpose
• The project is a secure cloud-based Linux computer system that leverages the E2B Desktop Sandbox. It is controlled by open-source LLMs, meaning that it comprises components that interface with various language model providers (e.g., Groq, OpenAI, Hugging Face) and manages interactions through configurable modules.
• The design is primarily focused on providing a secure sandboxed desktop environment that the user can control through keyboard, mouse, shell commands, and also view via live streaming. The system also supports dynamic configuration of LLMs and integration with external APIs.
File Structure
• At the top level, key files include “main.py”, which likely serves as the entry point into the program, and “pyproject.toml” for dependency management.
• The directory “os_computer_use” contains most of the core modules, such as:
• browser_script.py: Likely handles interactions coming from or output to a browser interface.
• config.py: Contains configuration and setup for LLM providers.
• grounding.py, llm_provider.py, providers.py: Modules for the integration with LLMs. This establishes abstraction over multiple LLM providers.
• logging.py: Responsible for logging system events.
• sandbox.py and sandbox_agent.py: Handle the secure sandbox environment and the agent operations within that environment.
• streaming.py: Likely deals with live display streaming of the desktop environment.
• templates directory with log.html: Provides front-end templates possibly for logging or UI rendering.
• The “tests” directory includes unit tests for critical modules like “llm_provider.py” and “sandbox_agent.py”.
Key Details from the README
• The README clearly explains the feature set—secure desktop sandbox, integration with multiple LLM providers, streaming display, and control via keyboard/mouse/shell commands.
• It outlines how to swap in/out providers by modifying config.py and providers.py, emphasizing the modularity and extensibility of the LLM integration.
• Additional external documentation is referenced, making the project design clear: a separation between core sandbox functionality and LLM-based decision making.
a. Main Components:
• Sandbox Environment:
• Modules: sandbox.py, sandbox_agent.py, streaming.py.
• Purpose: Managing the isolated cloud Linux instance, live display streaming, and agent control.
• LLM Integration Layer:
• Modules: llm_provider.py, grounding.py, providers.py, config.py.
• Purpose: Interfacing with various LLM providers, including support for vision and action tasks.
• User Interface Component:
• Components: browser_script.py and the web UI templates (log.html in “templates”).
• Purpose: Allows users to interact with the system, observe streamed output, and provide input.
• Core Controller/Entry Point:
• Module: main.py.
• Purpose: Boots up the application, initializes the environment, and orchestrates component interactions.
• Supporting Components:
• logging.py for system logging.
• Additional directories for assets (including pre-made architecture images) and configuration files (like pyproject.toml).
b. Relationships and Interactions:
• Main.py initializes and coordinates interactions between the sandbox, LLM integration, and the UI.
• The sandbox (sandbox.py/sandbox_agent.py) communicates with the streaming module to provide live display data to the user interface.
• The LLM integration layer (providers, config, and llm_provider modules) processes user inputs and system commands, routing them via configured third-party LLM APIs.
• The browser_script acts as a client side component that likely sends user commands and displays streamed results.
• Logging is captured and may be rendered using templates (log.html) and reported throughout the system.
c. Important Architectural Patterns and Design Principles:
• Modular Design: Clear separation of concerns between sandbox operations, LLM provider configurations, and UI interactions.
• Extensibility: The use of a providers module (and associated config) permits easy integration of new LLM providers.
• Secure Operation: Emphasis on sandboxing a cloud-based Linux environment ensures operational security.
• Event Flow and Streaming: A data flow from user actions (via browser script/UI) to LLM processing then resulting in streamed feedback.
d. Relevant Technologies, Frameworks, or Libraries:
• Python is the core language with version 3.10+ required.
• E2B Desktop Sandbox is integrated for secure desktop isolation.
• The system integrates multiple third-party LLM APIs (OpenAI, Groq, Anthropic, etc.).
• The use of Poetry for dependency management and environment configuration (pyproject.toml) is noted.
• Frontend web technologies may be implied via browser_script.py and HTML templates.