# asaripa3/computer-use-automation-system architecture

> Architecture diagram of the GitHub repository asaripa3/computer-use-automation-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/asaripa3/computer-use-automation-system
- Repository: https://github.com/asaripa3/computer-use-automation-system

## Overview

This repository is a computer-use automation system for legacy back-office applications. Its main workflow is: a model explores an application through a browser surface, records and compiles that work into a typed capability, and a deterministic replay engine executes the capability under policy, returning success, a business outcome, or a failure. Human handoff is an escalation path for steps requiring intervention. The README and source index establish the major layers and several dependencies; unsampled details are represented only where documented or referenced.

Main entry points: [`src/replay/engine.py`](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/engine.py) (Replay engine), [`src/explore/loop.py`](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/loop.py) (Discovery loop), [`src/explore/tools.py`](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/tools.py) (Discovery tools), [`src/sharebase/app.py`](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/app.py) (ShareBase console)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_target["Target application"]
  node_sharebase["ShareBase console<br/>[app.py]"]
  node_member_data[("Member data<br/>[data.py]")]
  node_session["Session handling<br/>[session.py]"]
  node_faults["Fault injection<br/>[faults.py]"]
end

subgraph group_surface["Surface interface"]
  node_surface_model["Surface protocol<br/>[model.py]"]
  node_browser["Browser surface<br/>[browser.py]"]
  node_surface_view["Page view<br/>[view.py]"]
end

subgraph group_contract["Capability contract"]
  node_capability["Capability schema<br/>[capability.py]"]
  node_locator["Locator ladder<br/>[locator.py]"]
  node_values["Typed values<br/>[values.py]"]
  node_derive["Contract derivation<br/>[derive.py]"]
  node_results["Run results<br/>[result.py]"]
end

subgraph group_runtime["Discovery and replay"]
  node_explore_loop["Discovery loop<br/>[loop.py]"]
  node_explore_model["Model adapter<br/>[model.py]"]
  node_explore_tools["Discovery tools<br/>[tools.py]"]
  node_recorder["Capability recorder<br/>[recorder.py]"]
  node_replay_engine["Replay engine<br/>[engine.py]"]
  node_resolution["Target resolution<br/>[resolve.py]"]
  node_evidence["Replay evidence<br/>[evidence.py]"]
end

subgraph group_governance["Safety and handoff"]
  node_allowlist["Allowlist policy<br/>[allowlist.py]"]
  node_risk["Risk gating<br/>[risk.py]"]
  node_redaction["Data redaction<br/>[redaction.py]"]
  node_handoff_control["Session control<br/>[control.py]"]
  node_intervention["Intervention request<br/>[intervention.py]"]
  node_operator_channel["Operator channel<br/>[operator.py]"]
  node_coordinator["Handoff coordinator<br/>[coordinator.py]"]
end

node_operator_user(("Caller or operator"))

node_operator_user -->|"starts discovery"| node_explore_loop
node_explore_loop -->|"queries model"| node_explore_model
node_explore_loop -->|"dispatches tools"| node_explore_tools
node_explore_tools -->|"acts through"| node_browser
node_browser -->|"implements"| node_surface_model
node_explore_tools -->|"checks actions"| node_allowlist
node_explore_tools -->|"records actions"| node_recorder
node_explore_loop -->|"drives recording"| node_recorder
node_recorder -->|"derives contract"| node_derive
node_derive -->|"uses observations"| node_surface_model
node_derive -->|"builds locators"| node_locator
node_capability -->|"contains locator schema"| node_locator
node_capability -->|"uses typed values"| node_values
node_operator_user -->|"invokes capability"| node_replay_engine
node_replay_engine -->|"executes contract"| node_capability
node_replay_engine -->|"resolves targets"| node_resolution
node_resolution -->|"uses derivation"| node_derive
node_resolution -->|"evaluates locators"| node_locator
node_replay_engine -->|"observes and acts"| node_surface_model
node_replay_engine -->|"checks destinations"| node_allowlist
node_replay_engine -->|"gates risky steps"| node_risk
node_replay_engine -->|"redacts run data"| node_redaction
node_replay_engine -->|"transfers control"| node_handoff_control
node_replay_engine -->|"records evidence"| node_evidence
node_replay_engine -->|"returns result"| node_results
node_handoff_control -->|"raises request"| node_intervention
node_coordinator -->|"coordinates control"| node_handoff_control
node_coordinator -->|"routes intervention"| node_operator_channel
node_operator_channel -->|"requests decision"| node_operator_user
node_sharebase -->|"reads member records"| node_member_data
node_sharebase -->|"checks session"| node_session
node_sharebase -->|"checks armed faults"| node_faults
node_browser -->|"drives application"| node_sharebase
node_surface_view -->|"formats observations"| node_surface_model
node_evidence -->|"redacts evidence"| node_redaction

click node_sharebase "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/app.py"
click node_member_data "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/data.py"
click node_session "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/session.py"
click node_faults "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/faults.py"
click node_surface_model "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/surface/model.py"
click node_browser "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/surface/browser.py"
click node_surface_view "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/surface/view.py"
click node_capability "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/capability.py"
click node_locator "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/locator.py"
click node_values "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/values.py"
click node_derive "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/derive.py"
click node_results "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/result.py"
click node_explore_loop "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/loop.py"
click node_explore_model "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/model.py"
click node_explore_tools "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/tools.py"
click node_recorder "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/recorder.py"
click node_replay_engine "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/engine.py"
click node_resolution "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/resolve.py"
click node_evidence "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/evidence.py"
click node_allowlist "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/policy/allowlist.py"
click node_risk "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/policy/risk.py"
click node_redaction "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/policy/redaction.py"
click node_handoff_control "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/control.py"
click node_intervention "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/intervention.py"
click node_operator_channel "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/operator.py"
click node_coordinator "https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/coordinator.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sharebase,node_member_data,node_session,node_faults toneBlue
class node_surface_model,node_browser,node_surface_view toneAmber
class node_capability,node_locator,node_values,node_derive,node_results toneMint
class node_explore_loop,node_explore_model,node_explore_tools,node_recorder,node_replay_engine,node_resolution,node_evidence toneRose
class node_allowlist,node_risk,node_redaction,node_handoff_control,node_intervention,node_operator_channel,node_coordinator,node_operator_user toneIndigo
```

## Components

### Target application

- [ShareBase console](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/app.py): `src/sharebase/app.py`
- [Member data](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/data.py): `src/sharebase/data.py`
- [Session handling](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/session.py): `src/sharebase/session.py`
- [Fault injection](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/sharebase/faults.py): `src/sharebase/faults.py`

### Surface interface

- [Surface protocol](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/surface/model.py): `src/surface/model.py`
- [Browser surface](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/surface/browser.py): `src/surface/browser.py`
- [Page view](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/surface/view.py): `src/surface/view.py`

### Capability contract

- [Capability schema](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/capability.py): `src/contract/capability.py`
- [Locator ladder](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/locator.py): `src/contract/locator.py`
- [Typed values](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/values.py): `src/contract/values.py`
- [Contract derivation](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/derive.py): `src/contract/derive.py`
- [Run results](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/contract/result.py): `src/contract/result.py`

### Discovery and replay

- [Discovery loop](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/loop.py): `src/explore/loop.py`
- [Model adapter](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/model.py): `src/explore/model.py`
- [Discovery tools](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/tools.py): `src/explore/tools.py`
- [Capability recorder](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/explore/recorder.py): `src/explore/recorder.py`
- [Replay engine](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/engine.py): `src/replay/engine.py`
- [Target resolution](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/resolve.py): `src/replay/resolve.py`
- [Replay evidence](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/replay/evidence.py): `src/replay/evidence.py`

### Safety and handoff

- [Allowlist policy](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/policy/allowlist.py): `src/policy/allowlist.py`
- [Risk gating](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/policy/risk.py): `src/policy/risk.py`
- [Data redaction](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/policy/redaction.py): `src/policy/redaction.py`
- [Session control](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/control.py): `src/handoff/control.py`
- [Intervention request](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/intervention.py): `src/handoff/intervention.py`
- [Operator channel](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/operator.py): `src/handoff/operator.py`
- [Handoff coordinator](https://github.com/asaripa3/computer-use-automation-system/blob/main/src/handoff/coordinator.py): `src/handoff/coordinator.py`

### Other components

- **Caller or operator**

## Connections

- Caller or operator → Discovery loop: starts discovery (evidence: `README.md`)
- Discovery loop → Model adapter: queries model (evidence: `src/explore/loop.py`)
- Discovery loop → Discovery tools: dispatches tools (evidence: `src/explore/loop.py`)
- Discovery tools → Browser surface: acts through (evidence: `src/explore/tools.py`)
- Browser surface → Surface protocol: implements (evidence: `src/surface/browser.py`)
- Discovery tools → Allowlist policy: checks actions (evidence: `src/explore/tools.py`)
- Discovery tools → Capability recorder: records actions (evidence: `src/explore/tools.py`)
- Discovery loop → Capability recorder: drives recording (evidence: `src/explore/loop.py`)
- Capability recorder → Contract derivation: derives contract (evidence: `src/explore/recorder.py`)
- Contract derivation → Surface protocol: uses observations (evidence: `src/contract/derive.py`)
- Contract derivation → Locator ladder: builds locators (evidence: `src/contract/derive.py`)
- Capability schema → Locator ladder: contains locator schema (evidence: `src/contract/capability.py`)
- Capability schema → Typed values: uses typed values (evidence: `src/contract/capability.py`)
- Caller or operator → Replay engine: invokes capability (evidence: `README.md`)
- Replay engine → Capability schema: executes contract (evidence: `src/replay/engine.py`)
- Replay engine → Target resolution: resolves targets (evidence: `src/replay/engine.py`)
- Target resolution → Contract derivation: uses derivation (evidence: `src/replay/resolve.py`)
- Target resolution → Locator ladder: evaluates locators (evidence: `src/replay/resolve.py`)
- Replay engine → Surface protocol: observes and acts (evidence: `src/replay/engine.py`)
- Replay engine → Allowlist policy: checks destinations (evidence: `src/replay/engine.py`)
- Replay engine → Risk gating: gates risky steps (evidence: `src/replay/engine.py`)
- Replay engine → Data redaction: redacts run data (evidence: `src/replay/engine.py`)
- Replay engine → Session control: transfers control (evidence: `src/replay/engine.py`)
- Replay engine → Replay evidence: records evidence (evidence: `src/replay/engine.py`)
- Replay engine → Run results: returns result (evidence: `src/replay/engine.py`)
- Session control → Intervention request: raises request (evidence: `src/handoff/control.py`)
- Handoff coordinator → Session control: coordinates control (evidence: `src/handoff/coordinator.py`)
- Handoff coordinator → Operator channel: routes intervention (evidence: `src/handoff/coordinator.py`)
- Operator channel → Caller or operator: requests decision (evidence: `README.md`)
- ShareBase console → Member data: reads member records (evidence: `src/sharebase/app.py`)
- ShareBase console → Session handling: checks session (evidence: `src/sharebase/app.py`)
- ShareBase console → Fault injection: checks armed faults (evidence: `src/sharebase/app.py`)
- Browser surface → ShareBase console: drives application (evidence: `README.md`)
- Page view → Surface protocol: formats observations (evidence: `src/surface/view.py`)
- Replay evidence → Data redaction: redacts evidence (evidence: `src/replay/evidence.py`)