How showlab/computer_use_ootb works
Project type and purpose
• This is a Python-based desktop GUI agent framework (“Computer Use OOTB”) that exposes a Gradio web interface for controlling a local or remote computer via natural-language commands. It combines:
• A front-end web interface (Gradio)
• A back-end orchestration layer (planner, actor, executor, loop)
• A toolkit for OS-level actions (screen capture, mouse/keyboard simulation, file edits)
• Integration with external LLM APIs (Anthropic Claude 3.5, OpenAI GPT-4o, Qwen)
• Optional local VLA models (ShowUI, UI-TARS)
File structure
• Top-level files:
• app.py: entry point that launches the Gradio interface
• computer_use_demo/: core logic
• install_tools/: scripts to install local models (ShowUI, UI-TARS)
• assets/: examples, logos, UI snapshots
• requirements.txt, README.md, .gitignore, LICENSE
• computer_use_demo/ contains:
• gui_agent/actor: low-level vision-language-action modules
• gui_agent/planner: high-level planning modules
• executor: modules that actually invoke the actor or API
• loop.py: main orchestration loop (intake user prompt → plan → act → feedback)
• remote_inference.py: remote planning support via SSH
• tools/: abstractions for OS operations (bash, screen capture, editing, logging)
• No persistent database; state flows through in-memory loop.
README for additional insights
• The README describes OOTB as a “full-stack” Desktop GUI agent supporting both API-based and local ML models.
• Highlights “Unified Model” vs separate Planner/Actor roles.
• Presents the user workflow:
user → Gradio UI → planner → actor → tools → OS actions → screenshot → planner …
• Blue rectangles for internal modules