# othersideai/self-operating-computer architecture

> Architecture diagram of the GitHub repository othersideai/self-operating-computer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/othersideai/self-operating-computer
- Repository: https://github.com/othersideai/self-operating-computer

## Overview

##### Main Components
- Command Line Interface (CLI) Entry Point:
– The project is executed by invoking the “operate” command (see files evaluate.py, operate/main.py, and operate/operate.py). Represent this as the entry point for user inputs.
- Configuration & Controller Module:
– The “operate/config.py” and “operate/exceptions.py” files suggest a component that handles configurations, mode selections, and error management.
- Model Integration Layer:
– In the “operate/models/” directory, you have modules such as “apis.py” and “prompts.py”. These manage the interactions between the framework and the various multimodal models (e.g., gpt-4o, o1, gemini-pro-vision, claude-3, qwen-vl, llava) with specific modes like “gpt-4-with-ocr” and “gpt-4-with-som.”
– The weights (in “operate/models/weights/best.pt”) indicate that for certain modes (like Set-of-Mark prompting), a pretrained model (YOLOv8 for button detection) is utilized.
- Utility Modules:
– The “operate/utils/” directory includes modules such as “ocr.py” (for optical character recognition), “screenshot.py” (for capturing screen content), “operating_system.py” (interacting with the OS for permissions and actions), “label.py”, “misc.py”, and “style.py”. These utilities provide supporting functionalities like data processing and OS interactions.
- External Dependencies & API Integrations:
– The README explains integration with external services (e.g., OpenAI, Google AI Studio, Anthropic, and the Ollama service). Represent an external integration layer where user-provided API keys enable communication with these external multimodal model services.

##### Relationships & Interactions
- User Input and Invocation:
- Mode Selection and Configuration:
– This component reads configuration details (like API keys and mode selections) and decides which “Model Integration Layer” to call.
- Data Flow between Modules:
– The “Model Integration Layer” communicates with external model APIs (using the details in “apis.py”, etc.), and then receives a response representing the AI’s decision.
- Action Execution:
– Finally, based on model decisions and processed data, the “Operating System” interface (using code in “operating_system.py”) issues system-level clicks or keyboard commands.

##### Architectural Patterns & Design Principles
- Modular Design: Separate components for configuration, model interactions, utilities, and OS actions.
- Extensible Architecture: Through clearly defined modules (for APIs, prompts, and weights), the system is designed to easily support additional multimodal models and modes.
- Abstraction Layers: The framework abstracts interactions with external services from its core operational logic.
- Error Handling: Use of “exceptions.py” ensures robust error management across modules.

## Diagram (Mermaid)

```mermaid
graph TD
    %% CLI / User Input
    subgraph "CLI / User Input"
        CLI1("evaluate.py"):::cli
        CLI2("operate/main.py"):::cli
        CLI3("operate/operate.py"):::cli
    end

    %% Configuration & Controller Module
    subgraph "Configuration & Controller Module"
        Config("Controller & Config"):::controller
        Exceptions("Exceptions"):::controller
    end

    %% Model Integration Layer
    subgraph "Model Integration Layer"
        APIs("APIs"):::model
        Prompts("Prompts"):::model
        Weights("YOLOv8 Weights"):::model
    end

    %% External AI Model Services
    subgraph "External AI Model Services"
        GPT4("GPT-4_API"):::external
        Google("Google_AI_Studio"):::external
        Anthropic("Anthropic_Service"):::external
        Ollama("Ollama_Service"):::external
    end

    %% Utility Modules
    subgraph "Utility Modules"
        OCR("OCR"):::utility
        Screenshot("Screenshot"):::utility
        OSInterface("OS Interface"):::utility
        Label("Label"):::utility
        Misc("Misc"):::utility
        Style("Style"):::utility
    end

    %% Relationships: User Input to Configuration
    CLI1 -->|"invoke"| Config
    CLI2 -->|"invoke"| Config
    CLI3 -->|"invoke"| Config

    %% Configuration to Exceptions (error handling)
    Config -->|"handles"| Exceptions

    %% Configuration triggers Model Integration
    Config -->|"selects_mode"| APIs
    Config -->|"selects_mode"| Prompts

    %% Model Integration internal dependency
    APIs -->|"uses_weights"| Weights

    %% Model Integration calls External Services
    APIs -->|"calls"| GPT4
    APIs -->|"calls"| Google
    APIs -->|"calls"| Anthropic
    APIs -->|"calls"| Ollama

    %% Utilities feeding visual data into Model Integration
    OCR -->|"provides_text"| APIs
    Screenshot -->|"captures_screen"| APIs

    %% Utilities assist Controller with additional info
    Label -->|"supplies_labels"| Config
    Misc -->|"provides_misc"| Config
    Style -->|"applies_style"| Config

    %% Controller delegates execution to OS Interface
    Config -->|"executes_commands"| OSInterface

    %% Click Events for CLI Entry Point
    click CLI1 "https://github.com/othersideai/self-operating-computer/blob/main/evaluate.py"
    click CLI2 "https://github.com/othersideai/self-operating-computer/blob/main/operate/main.py"
    click CLI3 "https://github.com/othersideai/self-operating-computer/blob/main/operate/operate.py"

    %% Click Events for Configuration & Controller Module
    click Config "https://github.com/othersideai/self-operating-computer/blob/main/operate/config.py"
    click Exceptions "https://github.com/othersideai/self-operating-computer/blob/main/operate/exceptions.py"

    %% Click Events for Model Integration Layer
    click APIs "https://github.com/othersideai/self-operating-computer/blob/main/operate/models/apis.py"
    click Prompts "https://github.com/othersideai/self-operating-computer/blob/main/operate/models/prompts.py"
    click Weights "https://github.com/othersideai/self-operating-computer/blob/main/operate/models/weights/best.pt"

    %% Click Events for Utility Modules
    click OCR "https://github.com/othersideai/self-operating-computer/blob/main/operate/utils/ocr.py"
    click Screenshot "https://github.com/othersideai/self-operating-computer/blob/main/operate/utils/screenshot.py"
    click OSInterface "https://github.com/othersideai/self-operating-computer/blob/main/operate/utils/operating_system.py"
    click Label "https://github.com/othersideai/self-operating-computer/blob/main/operate/utils/label.py"
    click Misc "https://github.com/othersideai/self-operating-computer/blob/main/operate/utils/misc.py"
    click Style "https://github.com/othersideai/self-operating-computer/blob/main/operate/utils/style.py"

classDef cli fill:#FFE0B2,stroke:#FB8C00,stroke-width:2px;
classDef controller fill:#BBDEFB,stroke:#1976D2,stroke-width:2px;
classDef model fill:#C8E6C9,stroke:#388E3C,stroke-width:2px;
classDef external fill:#FFCDD2,stroke:#D32F2F,stroke-width:2px;
classDef utility fill:#FFF9C4,stroke:#FBC02D,stroke-width:2px;
```
