# agent-sh/computer-use-linux architecture

> Architecture diagram of the GitHub repository agent-sh/computer-use-linux, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/agent-sh/computer-use-linux
- Repository: https://github.com/agent-sh/computer-use-linux

## Overview

computer-use-linux is a Linux desktop-control service for AI agents, exposed through MCP, a CLI, and Pi-native tools. The main workflow is to inspect desktop state (accessibility and windows), target a window or element, perform input actions, and optionally verify the result with a screenshot and updated tree. The README documents broad capabilities, while sampled source and the SOURCE INDEX confirm several internal calls; unsampled server and screenshot wiring is represented only at the documented level.

Main entry points: [`src/server.rs`](https://github.com/agent-sh/computer-use-linux/blob/main/src/server.rs) (MCP server), [`src/diagnostics.rs`](https://github.com/agent-sh/computer-use-linux/blob/main/src/diagnostics.rs) (Readiness checks), [`src/windowing/backends`](https://github.com/agent-sh/computer-use-linux/tree/main/src/windowing/backends) (Window backends), [`src/screenshot.rs`](https://github.com/agent-sh/computer-use-linux/blob/main/src/screenshot.rs) (Screenshot capture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Agent interfaces"]
  node_mcp["MCP server<br/>[server.rs]"]
  node_cli["CLI<br/>[cli.rs]"]
  node_pi["Pi tools<br/>[index.ts]"]
  node_npm["npm launcher"]
end

subgraph group_inspect["Desktop inspection"]
  node_accessibility["AT-SPI trees<br/>[atspi_tree.rs]"]
  node_windowing["Window targeting<br/>[mod.rs]"]
  node_registry["Backend registry<br/>[registry.rs]"]
  node_backends["Window backends"]
  node_terminal["Terminal context<br/>[terminal.rs]"]
end

subgraph group_control["Input and capture"]
  node_input["Input actions<br/>[remote_desktop.rs]"]
  node_ydotool["ydotool input<br/>[ydotool.rs]"]
  node_hyprinput["Hyprland input<br/>[hyprland_input.rs]"]
  node_screenshots["Screenshot capture<br/>[screenshot.rs]"]
  node_pixelflux["Pixel transforms<br/>[pixelflux.rs]"]
  node_indicator["Activity indicator<br/>[indicator.rs]"]
end

subgraph group_platform["Desktop integrations"]
  node_gnomeext["GNOME extension<br/>[extension.js]"]
  node_cosmic["COSMIC helper"]
  node_x11["X11 display<br/>[x11_display.rs]"]
end

subgraph group_operations["Readiness and support"]
  node_diagnostics["Readiness checks<br/>[diagnostics.rs]"]
  node_commands["Command runner<br/>[command_runner.rs]"]
end

node_agent(("AI agent"))

node_agent -->|"uses MCP"| node_mcp
node_agent -->|"uses CLI"| node_cli
node_agent -->|"uses native tools"| node_pi
node_agent -->|"installs CLI"| node_npm
node_mcp -.->|"exposes inspection"| node_accessibility
node_mcp -.->|"exposes targeting"| node_windowing
node_mcp -.->|"exposes actions"| node_input
node_mcp -.->|"exposes capture"| node_screenshots
node_mcp -.->|"exposes readiness"| node_diagnostics
node_cli -->|"provides doctor"| node_diagnostics
node_pi -->|"connects MCP"| node_mcp
node_windowing -->|"resolves targets"| node_registry
node_registry -->|"probes backends"| node_backends
node_backends -->|"enriches windows"| node_terminal
node_diagnostics -->|"checks windowing"| node_registry
node_diagnostics -->|"checks input"| node_ydotool
node_diagnostics -.->|"runs checks"| node_commands
node_input -->|"uses helper"| node_cosmic
node_input -.->|"uses input"| node_hyprinput
node_ydotool -.->|"runs commands"| node_commands
node_backends -.->|"runs commands"| node_commands
node_screenshots -.->|"transforms images"| node_pixelflux
node_screenshots -->|"captures X11"| node_x11
node_screenshots -->|"hides before capture"| node_indicator
node_backends -.->|"queries extension"| node_gnomeext

click node_mcp "https://github.com/agent-sh/computer-use-linux/blob/main/src/server.rs"
click node_cli "https://github.com/agent-sh/computer-use-linux/blob/main/src/cli.rs"
click node_pi "https://github.com/agent-sh/computer-use-linux/blob/main/pi/extension/index.ts"
click node_accessibility "https://github.com/agent-sh/computer-use-linux/blob/main/src/atspi_tree.rs"
click node_windowing "https://github.com/agent-sh/computer-use-linux/blob/main/src/windowing/mod.rs"
click node_registry "https://github.com/agent-sh/computer-use-linux/blob/main/src/windowing/registry.rs"
click node_backends "https://github.com/agent-sh/computer-use-linux/tree/main/src/windowing/backends"
click node_terminal "https://github.com/agent-sh/computer-use-linux/blob/main/src/terminal.rs"
click node_input "https://github.com/agent-sh/computer-use-linux/blob/main/src/remote_desktop.rs"
click node_ydotool "https://github.com/agent-sh/computer-use-linux/blob/main/src/ydotool.rs"
click node_hyprinput "https://github.com/agent-sh/computer-use-linux/blob/main/src/hyprland_input.rs"
click node_screenshots "https://github.com/agent-sh/computer-use-linux/blob/main/src/screenshot.rs"
click node_pixelflux "https://github.com/agent-sh/computer-use-linux/blob/main/src/pixelflux.rs"
click node_indicator "https://github.com/agent-sh/computer-use-linux/blob/main/src/indicator.rs"
click node_gnomeext "https://github.com/agent-sh/computer-use-linux/blob/main/gnome-shell-extension/computer-use-linux%40avifenesh.dev/extension.js"
click node_cosmic "https://github.com/agent-sh/computer-use-linux/blob/main/src/bin/computer-use-linux-cosmic.rs"
click node_x11 "https://github.com/agent-sh/computer-use-linux/blob/main/src/x11_display.rs"
click node_diagnostics "https://github.com/agent-sh/computer-use-linux/blob/main/src/diagnostics.rs"
click node_commands "https://github.com/agent-sh/computer-use-linux/blob/main/src/command_runner.rs"
click node_npm "https://github.com/agent-sh/computer-use-linux/blob/main/npm/bin/computer-use-linux.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mcp,node_cli,node_pi,node_npm toneBlue
class node_accessibility,node_windowing,node_registry,node_backends,node_terminal toneAmber
class node_input,node_ydotool,node_hyprinput,node_screenshots,node_pixelflux,node_indicator toneMint
class node_gnomeext,node_cosmic,node_x11 toneRose
class node_diagnostics,node_commands,node_agent toneIndigo
```

## Components

### Agent interfaces

- [MCP server](https://github.com/agent-sh/computer-use-linux/blob/main/src/server.rs): `src/server.rs`
- [CLI](https://github.com/agent-sh/computer-use-linux/blob/main/src/cli.rs): `src/cli.rs`
- [Pi tools](https://github.com/agent-sh/computer-use-linux/blob/main/pi/extension/index.ts): `pi/extension/index.ts`
- [npm launcher](https://github.com/agent-sh/computer-use-linux/blob/main/npm/bin/computer-use-linux.js): `npm/bin/computer-use-linux.js`

### Desktop inspection

- [AT-SPI trees](https://github.com/agent-sh/computer-use-linux/blob/main/src/atspi_tree.rs): `src/atspi_tree.rs`
- [Window targeting](https://github.com/agent-sh/computer-use-linux/blob/main/src/windowing/mod.rs): `src/windowing/mod.rs`
- [Backend registry](https://github.com/agent-sh/computer-use-linux/blob/main/src/windowing/registry.rs): `src/windowing/registry.rs`
- [Window backends](https://github.com/agent-sh/computer-use-linux/tree/main/src/windowing/backends): `src/windowing/backends`
- [Terminal context](https://github.com/agent-sh/computer-use-linux/blob/main/src/terminal.rs): `src/terminal.rs`

### Input and capture

- [Input actions](https://github.com/agent-sh/computer-use-linux/blob/main/src/remote_desktop.rs): `src/remote_desktop.rs`
- [ydotool input](https://github.com/agent-sh/computer-use-linux/blob/main/src/ydotool.rs): `src/ydotool.rs`
- [Hyprland input](https://github.com/agent-sh/computer-use-linux/blob/main/src/hyprland_input.rs): `src/hyprland_input.rs`
- [Screenshot capture](https://github.com/agent-sh/computer-use-linux/blob/main/src/screenshot.rs): `src/screenshot.rs`
- [Pixel transforms](https://github.com/agent-sh/computer-use-linux/blob/main/src/pixelflux.rs): `src/pixelflux.rs`
- [Activity indicator](https://github.com/agent-sh/computer-use-linux/blob/main/src/indicator.rs): `src/indicator.rs`

### Desktop integrations

- [GNOME extension](https://github.com/agent-sh/computer-use-linux/blob/main/gnome-shell-extension/computer-use-linux%40avifenesh.dev/extension.js): `gnome-shell-extension/computer-use-linux@avifenesh.dev/extension.js`
- [COSMIC helper](https://github.com/agent-sh/computer-use-linux/blob/main/src/bin/computer-use-linux-cosmic.rs): `src/bin/computer-use-linux-cosmic.rs`
- [X11 display](https://github.com/agent-sh/computer-use-linux/blob/main/src/x11_display.rs): `src/x11_display.rs`

### Readiness and support

- [Readiness checks](https://github.com/agent-sh/computer-use-linux/blob/main/src/diagnostics.rs): `src/diagnostics.rs`
- [Command runner](https://github.com/agent-sh/computer-use-linux/blob/main/src/command_runner.rs): `src/command_runner.rs`

### Other components

- **AI agent**

## Connections

- AI agent → MCP server: uses MCP
- AI agent → CLI: uses CLI
- AI agent → Pi tools: uses native tools
- AI agent → npm launcher: installs CLI
- MCP server → AT-SPI trees: exposes inspection
- MCP server → Window targeting: exposes targeting
- MCP server → Input actions: exposes actions
- MCP server → Screenshot capture: exposes capture
- MCP server → Readiness checks: exposes readiness
- CLI → Readiness checks: provides doctor
- Pi tools → MCP server: connects MCP (evidence: `pi/extension/index.ts`)
- Window targeting → Backend registry: resolves targets (evidence: `src/windowing/mod.rs`)
- Backend registry → Window backends: probes backends (evidence: `src/windowing/registry.rs`)
- Window backends → Terminal context: enriches windows (evidence: `src/windowing/backends/gnome.rs`)
- Readiness checks → Backend registry: checks windowing (evidence: `src/diagnostics.rs`)
- Readiness checks → ydotool input: checks input (evidence: `src/diagnostics.rs`)
- Readiness checks → Command runner: runs checks (evidence: `src/diagnostics.rs`)
- Input actions → COSMIC helper: uses helper (evidence: `src/remote_desktop.rs`)
- Input actions → Hyprland input: uses input
- ydotool input → Command runner: runs commands (evidence: `src/ydotool.rs`)
- Window backends → Command runner: runs commands (evidence: `src/windowing/backends/niri.rs`)
- Screenshot capture → Pixel transforms: transforms images
- Screenshot capture → X11 display: captures X11
- Screenshot capture → Activity indicator: hides before capture
- Window backends → GNOME extension: queries extension