# mithra2000/computer-use-automation architecture

> Architecture diagram of the GitHub repository mithra2000/computer-use-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mithra2000/computer-use-automation
- Repository: https://github.com/mithra2000/computer-use-automation

## Overview

This is a computer-use automation system for learning and replaying a read-only banking lookup. A user starts discovery or replay through the CLI: discovery observes the banking UI, obtains model proposals, authorizes and executes actions, verifies the member and savings data, then saves a typed capability; replay validates and executes that capability without model decisions. Policy enforcement, browser access, evidence capture, and optional human takeover support both paths. Some subsystem links reflect the README because their implementation was not excerpted.

Main entry points: [`automation/replay/engine.py`](https://github.com/mithra2000/computer-use-automation/blob/main/automation/replay/engine.py) (Replay engine), [`automation/cli.py`](https://github.com/mithra2000/computer-use-automation/blob/main/automation/cli.py) (Automation CLI), [`automation/discovery/agent.py`](https://github.com/mithra2000/computer-use-automation/blob/main/automation/discovery/agent.py) (Discovery agent), [`automation/surface/browser.py`](https://github.com/mithra2000/computer-use-automation/blob/main/automation/surface/browser.py) (Browser adapter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Run orchestration"]
  node_cli["Automation CLI<br/>[cli.py]"]
  node_server["Local server<br/>[local_server.py]"]
  node_result["Validated result<br/>[contracts.py]"]
end

subgraph group_learning["Capability discovery"]
  node_discovery["Discovery agent<br/>[agent.py]"]
  node_provider["Model provider<br/>[providers.py]"]
  node_builder["Artifact builder<br/>[builder.py]"]
  node_capability["Versioned capability<br/>[v1.json]"]
end

subgraph group_execution["Controlled execution"]
  node_replay["Replay engine<br/>[engine.py]"]
  node_surface["Browser adapter<br/>[browser.py]"]
end

subgraph group_support["Safety and evidence"]
  node_contracts["Typed contracts<br/>[contracts.py]"]
  node_policy["Policy engine<br/>[engine.py]"]
  node_evidence["Evidence recorder<br/>[recorder.py]"]
  node_handoff["Human handoff<br/>[manager.py]"]
end

subgraph group_bank["Banking application"]
  node_bankapp["LedgerDesk app<br/>[app.py]"]
end

node_user(("Operator"))
node_model(("LLM service"))
node_operator(("Human operator"))

node_user -->|"starts run"| node_cli
node_cli -->|"starts discovery"| node_discovery
node_cli -->|"creates provider"| node_provider
node_provider -->|"requests proposals"| node_model
node_discovery -->|"requests decisions"| node_provider
node_discovery -->|"observes and acts"| node_surface
node_discovery -->|"builds capability"| node_builder
node_builder -->|"saves artifact"| node_capability
node_cli -->|"starts replay"| node_replay
node_cli -->|"starts optional app"| node_server
node_server -->|"launches app"| node_bankapp
node_replay -->|"loads artifact"| node_capability
node_replay -->|"executes steps"| node_surface
node_replay -->|"checks authorization"| node_policy
node_surface -->|"authorizes browser actions"| node_policy
node_surface -->|"interacts through UI"| node_bankapp
node_discovery -->|"records run"| node_evidence
node_replay -->|"records run"| node_evidence
node_replay -->|"requests intervention"| node_handoff
node_handoff -->|"requests takeover"| node_operator
node_handoff -->|"resumes session"| node_surface
node_cli -->|"returns run result"| node_result
node_discovery -->|"validates proposals"| node_contracts
node_builder -->|"constructs capability"| node_contracts
node_replay -->|"validates result"| node_contracts

click node_cli "https://github.com/mithra2000/computer-use-automation/blob/main/automation/cli.py"
click node_server "https://github.com/mithra2000/computer-use-automation/blob/main/automation/local_server.py"
click node_contracts "https://github.com/mithra2000/computer-use-automation/blob/main/automation/models/contracts.py"
click node_discovery "https://github.com/mithra2000/computer-use-automation/blob/main/automation/discovery/agent.py"
click node_provider "https://github.com/mithra2000/computer-use-automation/blob/main/automation/discovery/providers.py"
click node_builder "https://github.com/mithra2000/computer-use-automation/blob/main/automation/artifacts/builder.py"
click node_capability "https://github.com/mithra2000/computer-use-automation/blob/main/capabilities/get_savings_balance/v1.json"
click node_replay "https://github.com/mithra2000/computer-use-automation/blob/main/automation/replay/engine.py"
click node_surface "https://github.com/mithra2000/computer-use-automation/blob/main/automation/surface/browser.py"
click node_policy "https://github.com/mithra2000/computer-use-automation/blob/main/automation/policy/engine.py"
click node_evidence "https://github.com/mithra2000/computer-use-automation/blob/main/automation/evidence/recorder.py"
click node_handoff "https://github.com/mithra2000/computer-use-automation/blob/main/automation/handoff/manager.py"
click node_bankapp "https://github.com/mithra2000/computer-use-automation/blob/main/mock_bank/app.py"
click node_result "https://github.com/mithra2000/computer-use-automation/blob/main/automation/models/contracts.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_server,node_result toneBlue
class node_discovery,node_provider,node_builder,node_capability toneAmber
class node_replay,node_surface toneMint
class node_contracts,node_policy,node_evidence,node_handoff toneRose
class node_bankapp,node_user,node_model,node_operator toneIndigo
```

## Components

### Run orchestration

- [Automation CLI](https://github.com/mithra2000/computer-use-automation/blob/main/automation/cli.py): `automation/cli.py`
- [Local server](https://github.com/mithra2000/computer-use-automation/blob/main/automation/local_server.py): `automation/local_server.py`
- [Validated result](https://github.com/mithra2000/computer-use-automation/blob/main/automation/models/contracts.py): `automation/models/contracts.py`

### Capability discovery

- [Discovery agent](https://github.com/mithra2000/computer-use-automation/blob/main/automation/discovery/agent.py): `automation/discovery/agent.py`
- [Model provider](https://github.com/mithra2000/computer-use-automation/blob/main/automation/discovery/providers.py): `automation/discovery/providers.py`
- [Artifact builder](https://github.com/mithra2000/computer-use-automation/blob/main/automation/artifacts/builder.py): `automation/artifacts/builder.py`
- [Versioned capability](https://github.com/mithra2000/computer-use-automation/blob/main/capabilities/get_savings_balance/v1.json): `capabilities/get_savings_balance/v1.json`

### Controlled execution

- [Replay engine](https://github.com/mithra2000/computer-use-automation/blob/main/automation/replay/engine.py): `automation/replay/engine.py`
- [Browser adapter](https://github.com/mithra2000/computer-use-automation/blob/main/automation/surface/browser.py): `automation/surface/browser.py`

### Safety and evidence

- [Typed contracts](https://github.com/mithra2000/computer-use-automation/blob/main/automation/models/contracts.py): `automation/models/contracts.py`
- [Policy engine](https://github.com/mithra2000/computer-use-automation/blob/main/automation/policy/engine.py): `automation/policy/engine.py`
- [Evidence recorder](https://github.com/mithra2000/computer-use-automation/blob/main/automation/evidence/recorder.py): `automation/evidence/recorder.py`
- [Human handoff](https://github.com/mithra2000/computer-use-automation/blob/main/automation/handoff/manager.py): `automation/handoff/manager.py`

### Banking application

- [LedgerDesk app](https://github.com/mithra2000/computer-use-automation/blob/main/mock_bank/app.py): `mock_bank/app.py`

### Other components

- **Operator**
- **LLM service**
- **Human operator**

## Connections

- Operator → Automation CLI: starts run (evidence: `README.md`)
- Automation CLI → Discovery agent: starts discovery (evidence: `automation/cli.py`)
- Automation CLI → Model provider: creates provider (evidence: `automation/cli.py`)
- Model provider → LLM service: requests proposals (evidence: `README.md`)
- Discovery agent → Model provider: requests decisions (evidence: `automation/discovery/agent.py`)
- Discovery agent → Browser adapter: observes and acts (evidence: `automation/discovery/agent.py`)
- Discovery agent → Artifact builder: builds capability (evidence: `automation/discovery/agent.py`)
- Artifact builder → Versioned capability: saves artifact (evidence: `automation/artifacts/builder.py`)
- Automation CLI → Replay engine: starts replay (evidence: `automation/cli.py`)
- Automation CLI → Local server: starts optional app (evidence: `automation/cli.py`)
- Local server → LedgerDesk app: launches app (evidence: `automation/local_server.py`)
- Replay engine → Versioned capability: loads artifact (evidence: `README.md`)
- Replay engine → Browser adapter: executes steps (evidence: `automation/replay/engine.py`)
- Replay engine → Policy engine: checks authorization (evidence: `automation/replay/engine.py`)
- Browser adapter → Policy engine: authorizes browser actions (evidence: `automation/surface/browser.py`)
- Browser adapter → LedgerDesk app: interacts through UI (evidence: `README.md`)
- Discovery agent → Evidence recorder: records run (evidence: `automation/discovery/agent.py`)
- Replay engine → Evidence recorder: records run (evidence: `automation/replay/engine.py`)
- Replay engine → Human handoff: requests intervention (evidence: `automation/replay/engine.py`)
- Human handoff → Human operator: requests takeover (evidence: `automation/handoff/manager.py`)
- Human handoff → Browser adapter: resumes session (evidence: `automation/handoff/manager.py`)
- Automation CLI → Validated result: returns run result (evidence: `automation/cli.py`)
- Discovery agent → Typed contracts: validates proposals (evidence: `automation/discovery/agent.py`)
- Artifact builder → Typed contracts: constructs capability (evidence: `automation/artifacts/builder.py`)
- Replay engine → Typed contracts: validates result (evidence: `automation/replay/engine.py`)
