How clevrr-ai/clevrr-computer works
Project type and purpose
• This is a full-stack automation agent (desktop/CLI application) named “Clevrr Computer.”
• It orchestrates multi-modal AI interactions (screen capture + LLM chains of thought) and executes OS-level actions (mouse, keyboard) via PyAutoGUI.
File structure
• Top level:
• main.py – CLI entry point and optional floating UI
• requirements.txt / .env_dev – dependency list and configuration template
• examples/ – demo artifacts (screenshots, GIFs)
• utils/ package:
• utils/agent.py – core agent orchestration (task ingestion, chain of thought, tool invocation)
• utils/prompt.py – prompt templates & injection logic
• utils/tools.py – tool implementations (get_screen_info, PythonREPLAst)
• utils/contants.py – configuration constants
README for additional insights
• Describes a continuous loop: capture screenshot → grid-based coordinate mapping → multimodal LLM understanding → chain of thought → tool-driven OS actions.
• External dependencies: Azure OpenAI or Google Gemini APIs.
Main components
a. User Interface & CLI (main.py + TKinter floating UI)
b. Configuration Loader (.env / .env_dev)
c. Agent Core (utils/agent.py)
d. Prompt Manager (utils/prompt.py)
e. Tooling Layer (utils/tools.py)
• get_screen_info tool
• PythonREPLAst tool
f. PyAutoGUI Library (executes actual mouse/keyboard/screenshot commands)
g. External LLM Services (Azure OpenAI, Google Gemini)
h. Operating System / Display Subsystem (screen pixels, windowing)
Relationships and interactions
• User invokes main.py → Configuration Loader reads .env → initializes Agent Core + UI
• Agent Core spins event loop:
1. Calls get_screen_info tool → uses PyAutoGUI to screenshot → grid-mapper → returns structured screen data to Agent
2. Agent Core calls Prompt Manager to build LLM prompt (chain of thought + screen data)
3. Sends prompt to External LLM Service → receives response (next action)
4. If action requires code execution, Agent calls PythonREPLAst tool → uses PyAutoGUI to simulate inputs
5. Loop continues until task completion or error
• UI displays status and captures user flags (e.g., float-ui on/off)
Architectural patterns and design principles
• Modular layering: UI/CLI layer, orchestration layer, tooling layer, external services
• Single responsibility modules: agent orchestration vs prompt generation vs tool execution
• Dependency inversion: core Agent relies on abstract tool interfaces, not directly on PyAutoGUI
Elements
• Clear labels: module names, tool names, API names