How agent-sh/computer-use-linux works
computer-use-linux is a Linux desktop-control service for AI agents, exposed through MCP, a CLI, and Pi-native tools. The main workflow is to inspect desktop state (accessibility and windows), target a window or element, perform input actions, and optionally verify the result with a screenshot and updated tree. The README documents broad capabilities, while sampled source and the SOURCE INDEX confirm several internal calls; unsampled server and screenshot wiring is represented only at the documented level.
Main entry points: server.rs MCP server · diagnostics.rs Readiness checks · backends Window backends · screenshot.rs Screenshot capture
Components 21
Agent interfaces
- MCP server
src/server.rs - CLI
src/cli.rs - Pi tools
pi/extension/index.ts - npm launcher
npm/bin/computer-use-linux.js
Desktop inspection
- AT-SPI trees
src/atspi_tree.rs - Window targeting
src/windowing/mod.rs - Backend registry
src/windowing/registry.rs - Window backends
src/windowing/backends - Terminal context
src/terminal.rs
Input and capture
- Input actions
src/remote_desktop.rs - ydotool input
src/ydotool.rs - Hyprland input
src/hyprland_input.rs - Screenshot capture
src/screenshot.rs - Pixel transforms
src/pixelflux.rs - Activity indicator
src/indicator.rs
Desktop integrations
- GNOME extension
gnome-shell-extension/computer-use-linux@avifenesh.dev/extension.js - COSMIC helper
src/bin/computer-use-linux-cosmic.rs - X11 display
src/x11_display.rs
Readiness and support
- Readiness checks
src/diagnostics.rs - Command runner
src/command_runner.rs
Other components
- AI agent
Connections 25
- AI agent → MCP server: uses MCP
- AI agent → CLI: uses CLI
- AI agent → Pi tools: uses native tools
- AI agent → npm launcher: installs CLI
- MCP server → AT-SPI trees: exposes inspection
- MCP server → Window targeting: exposes targeting
- MCP server → Input actions: exposes actions
- MCP server → Screenshot capture: exposes capture
- MCP server → Readiness checks: exposes readiness
- CLI → Readiness checks: provides doctor
- Pi tools → MCP server: connects MCP
pi/extension/index.ts - Window targeting → Backend registry: resolves targets
src/windowing/mod.rs - Backend registry → Window backends: probes backends
src/windowing/registry.rs - Window backends → Terminal context: enriches windows
src/windowing/backends/gnome.rs - Readiness checks → Backend registry: checks windowing
src/diagnostics.rs - Readiness checks → ydotool input: checks input
src/diagnostics.rs - Readiness checks → Command runner: runs checks
src/diagnostics.rs - Input actions → COSMIC helper: uses helper
src/remote_desktop.rs - Input actions → Hyprland input: uses input
- ydotool input → Command runner: runs commands
src/ydotool.rs - Window backends → Command runner: runs commands
src/windowing/backends/niri.rs - Screenshot capture → Pixel transforms: transforms images
- Screenshot capture → X11 display: captures X11
- Screenshot capture → Activity indicator: hides before capture
- Window backends → GNOME extension: queries extension