How othersideai/self-operating-computer works
Main Components
• Command Line Interface (CLI) Entry Point:
– The project is executed by invoking the “operate” command (see files evaluate.py, operate/main.py, and operate/operate.py). Represent this as the entry point for user inputs.
• Configuration & Controller Module:
– The “operate/config.py” and “operate/exceptions.py” files suggest a component that handles configurations, mode selections, and error management.
• Model Integration Layer:
– In the “operate/models/” directory, you have modules such as “apis.py” and “prompts.py”. These manage the interactions between the framework and the various multimodal models (e.g., gpt-4o, o1, gemini-pro-vision, claude-3, qwen-vl, llava) with specific modes like “gpt-4-with-ocr” and “gpt-4-with-som.”
– The weights (in “operate/models/weights/best.pt”) indicate that for certain modes (like Set-of-Mark prompting), a pretrained model (YOLOv8 for button detection) is utilized.
• Utility Modules:
– The “operate/utils/” directory includes modules such as “ocr.py” (for optical character recognition), “screenshot.py” (for capturing screen content), “operating_system.py” (interacting with the OS for permissions and actions), “label.py”, “misc.py”, and “style.py”. These utilities provide supporting functionalities like data processing and OS interactions.
• External Dependencies & API Integrations:
– The README explains integration with external services (e.g., OpenAI, Google AI Studio, Anthropic, and the Ollama service). Represent an external integration layer where user-provided API keys enable communication with these external multimodal model services.
Relationships & Interactions
• User Input and Invocation:
• Mode Selection and Configuration:
– This component reads configuration details (like API keys and mode selections) and decides which “Model Integration Layer” to call.
• Data Flow between Modules:
– The “Model Integration Layer” communicates with external model APIs (using the details in “apis.py”, etc.), and then receives a response representing the AI’s decision.
• Action Execution:
– Finally, based on model decisions and processed data, the “Operating System” interface (using code in “operating_system.py”) issues system-level clicks or keyboard commands.
Architectural Patterns & Design Principles
• Modular Design: Separate components for configuration, model interactions, utilities, and OS actions.
• Extensible Architecture: Through clearly defined modules (for APIs, prompts, and weights), the system is designed to easily support additional multimodal models and modes.
• Abstraction Layers: The framework abstracts interactions with external services from its core operational logic.
• Error Handling: Use of “exceptions.py” ensures robust error management across modules.