How ambersahdev/open-interface works
Project type and purpose
• Open Interface is a Python-based desktop application (full‑stack local app) that lets you “self‑drive” your computer by converting user requests into GUI automation steps via an LLM backend (e.g., GPT‑4o, Gemini).
• It captures screenshots, sends them + user goals to the LLM, receives instructions, interprets them into elementary actions, and executes them via simulated keyboard/mouse events.
File structure
• Top level:
• app/ – core application code
• assets/ – images, demo scripts
• build.py, requirements.txt – build and dependency definitions
• tests/ – test suite
• app/ breakdown:
• app.py – application entry point (initializes UI + Core)
• ui.py – GUI layer (Settings, Start/Stop, display)
• core.py – orchestrates the pipeline (screenshot → LLM → interpreter → executer)
• llm.py – LLM API client wrapper (OpenAI, Gemini, custom)
• interpreter.py – parses LLM output into actionable instructions
• models/ – model‑specific adapters and factory (plugin pattern)
• utils/ – OS‑specific helpers (screen capture, input simulation, settings persistence)
• resources/ – static context files for prompts
• version.py – versioning
README insights
• It describes the high‑level loop: screenshot + goal → LLM → instructions → execute → screenshot again.
Main components & interactions
a. Components
1. Frontend (UI Layer): app/ui.py
2. Core Controller: app/core.py
3. LLM Backend: External service (OpenAI GPT‑4o, Google Gemini, etc.) via app/llm.py
4. Interpreter: app/interpreter.py
5. Executer (Executor): low‑level GUI automation via app/utils/screen.py + local_info.py
6. Model Plugins: app/models/ (factory + adapters)
7. Resources & Settings: app/resources/ + app/utils/settings.py
8. Support Scripts: build.py, assets/mov_to_2x_mov_and_gif.py
9. Tests: tests/simple_test.py
10. External OS APIs: Screen capture, keyboard/mouse injection, Accessibility & Screen‑Recording permissions
b. Relationships & Data Flow
• User → UI: enters command, sees live logs/screenshots
• UI → Core: “Start session” + user goal
• Core → Utils (screen): capture screenshot
• Core → LLM: send screenshot + goal → receive action plan
• Core → Interpreter: parse LLM plan → atomic actions
• Interpreter → Executer: invoke utilities for mouse/keyboard
• Executer → OS APIs: perform actual events
• Loop: after actions, capture new screenshot → back to Core
• Settings UI ↔ Utils/settings: read/write API keys, model choices
• Core ↔ Models factory: choose correct model adapter
c. Architectural patterns
• MVC‑like separation: UI / Controller (Core) / Execution engine
• Plugin pattern for LLM/model adapters (factory + individual model classes)
• Pipeline pattern for processing: capture → plan → interpret → execute → repeat
d. Key technologies & libraries
• Python 3.12
• tkinter or similar for GUI (in app/ui.py)
• OpenAI / Google Gemini REST APIs (in app/llm.py)
• pyautogui or native OS calls (in app/utils/screen.py, local_info.py)
• Virtual environments, pytest for testing