# hithaishisurendra/computer-use-automation architecture

> Architecture diagram of the GitHub repository hithaishisurendra/computer-use-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hithaishisurendra/computer-use-automation
- Repository: https://github.com/hithaishisurendra/computer-use-automation

## Overview

This repository automates a hostile back-office UI: an LLM-led discovery run records a typed capability, then a separate deterministic replay executes it without a model. The product also supports tenant overlays, outcome classification, and human takeover of a live session. The graph follows documented entry points and source-index references; several implementation details are unsampled, so uncertain links are omitted rather than inferred.

Main entry points: [`replay/engine.py`](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/engine.py) (Replay orchestration), [`discovery/loop.py`](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/loop.py) (Observe-decide-act loop), [`discovery/run.py`](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/run.py) (Discovery entry), [`replay/run.py`](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/run.py) (Replay entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_target["CoreServ target"]
  node_core_app["CoreServ web app<br/>[main.py]"]
  node_core_data[("Member data<br/>[data.py]")]
  node_core_faults["Fault controls<br/>[faults.py]"]
end

subgraph group_discovery["Discovery"]
  node_discovery_run["Discovery entry<br/>[run.py]"]
  node_discovery_loop["Observe-decide-act loop<br/>[loop.py]"]
  node_model["Model providers<br/>[model.py]"]
  node_prompts["Discovery prompts<br/>[prompts.py]"]
  node_perception["Accessibility perception<br/>[tree.py]"]
  node_labeling["Label augmentation<br/>[labeling.py]"]
  node_recorder["Capability recorder<br/>[recorder.py]"]
end

subgraph group_contract["Capability contract"]
  node_artifact["Capability artifacts"]
  node_schema["Artifact schema<br/>[schema.py]"]
  node_loader["Artifact and overlay loader<br/>[loader.py]"]
  node_validation["Input and auth validation<br/>[validate.py]"]
  node_redaction["Sensitive data redaction<br/>[redaction.py]"]
end

subgraph group_replay["Deterministic replay"]
  node_replay_run["Replay entry<br/>[run.py]"]
  node_engine["Replay orchestration<br/>[engine.py]"]
  node_executor["Browser action executor<br/>[executor.py]"]
  node_resolver["Element resolver<br/>[resolver.py]"]
  node_checkpoints["Step checkpoints<br/>[checkpoints.py]"]
  node_classification["Outcome classification<br/>[classify.py]"]
  node_result["Replay result<br/>[result.py]"]
end

subgraph group_escalation["Human escalation"]
  node_escalation_session["Session ownership<br/>[session.py]"]
  node_escalation_request["Intervention request<br/>[request.py]"]
  node_capture["Handoff activity capture<br/>[capture.py]"]
end

node_operator(("Calling operator"))
node_human_operator(("Human operator"))

node_operator -->|"starts discovery"| node_discovery_run
node_operator -->|"requests replay"| node_replay_run
node_operator -->|"uses console"| node_core_app
node_core_app -->|"reads member data"| node_core_data
node_core_app -->|"checks fault flags"| node_core_faults
node_discovery_run -->|"starts loop"| node_discovery_loop
node_discovery_run -->|"selects provider"| node_model
node_discovery_run -->|"creates recorder"| node_recorder
node_discovery_run -->|"resolves credentials"| node_loader
node_discovery_loop -->|"requests decisions"| node_model
node_discovery_loop -->|"uses prompts"| node_prompts
node_discovery_loop -->|"observes page"| node_perception
node_perception -->|"augments labels"| node_labeling
node_discovery_loop -->|"executes actions"| node_executor
node_discovery_loop -->|"records run"| node_recorder
node_recorder -->|"builds artifact"| node_schema
node_discovery_run -->|"writes artifact"| node_artifact
node_artifact -->|"supplies capability"| node_replay_run
node_replay_run -->|"loads capability"| node_loader
node_loader -->|"parses artifact"| node_schema
node_replay_run -->|"runs replay"| node_engine
node_engine -->|"validates inputs"| node_validation
node_engine -->|"executes steps"| node_executor
node_executor -->|"resolves elements"| node_resolver
node_engine -->|"captures page state"| node_perception
node_engine -->|"evaluates checkpoints"| node_checkpoints
node_engine -->|"classifies outcome"| node_classification
node_engine -->|"produces result"| node_result
node_engine -->|"transfers control"| node_escalation_session
node_engine -->|"requests intervention"| node_escalation_request
node_engine -->|"captures handoff"| node_capture
node_discovery_loop -->|"supports takeover"| node_escalation_session
node_discovery_loop -->|"requests intervention"| node_escalation_request
node_human_operator -->|"receives request"| node_escalation_request
node_human_operator -->|"takes control"| node_escalation_session
node_replay_run -->|"uses safe evidence"| node_redaction
node_artifact -->|"provides tenant overlay"| node_loader

click node_core_app "https://github.com/hithaishisurendra/computer-use-automation/blob/main/coreserv/main.py"
click node_core_data "https://github.com/hithaishisurendra/computer-use-automation/blob/main/coreserv/data.py"
click node_core_faults "https://github.com/hithaishisurendra/computer-use-automation/blob/main/coreserv/faults.py"
click node_discovery_run "https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/run.py"
click node_discovery_loop "https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/loop.py"
click node_model "https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/model.py"
click node_prompts "https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/prompts.py"
click node_perception "https://github.com/hithaishisurendra/computer-use-automation/blob/main/perception/tree.py"
click node_labeling "https://github.com/hithaishisurendra/computer-use-automation/blob/main/perception/labeling.py"
click node_recorder "https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/recorder.py"
click node_artifact "https://github.com/hithaishisurendra/computer-use-automation/tree/main/capabilities"
click node_schema "https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/schema.py"
click node_loader "https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/loader.py"
click node_validation "https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/validate.py"
click node_redaction "https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/redaction.py"
click node_replay_run "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/run.py"
click node_engine "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/engine.py"
click node_executor "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/executor.py"
click node_resolver "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/resolver.py"
click node_checkpoints "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/checkpoints.py"
click node_classification "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/classify.py"
click node_result "https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/result.py"
click node_escalation_session "https://github.com/hithaishisurendra/computer-use-automation/blob/main/escalation/session.py"
click node_escalation_request "https://github.com/hithaishisurendra/computer-use-automation/blob/main/escalation/request.py"
click node_capture "https://github.com/hithaishisurendra/computer-use-automation/blob/main/escalation/capture.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_core_app,node_core_data,node_core_faults toneBlue
class node_discovery_run,node_discovery_loop,node_model,node_prompts,node_perception,node_labeling,node_recorder toneAmber
class node_artifact,node_schema,node_loader,node_validation,node_redaction toneMint
class node_replay_run,node_engine,node_executor,node_resolver,node_checkpoints,node_classification,node_result toneRose
class node_escalation_session,node_escalation_request,node_capture,node_operator,node_human_operator toneIndigo
```

## Components

### CoreServ target

- [CoreServ web app](https://github.com/hithaishisurendra/computer-use-automation/blob/main/coreserv/main.py): `coreserv/main.py`
- [Member data](https://github.com/hithaishisurendra/computer-use-automation/blob/main/coreserv/data.py): `coreserv/data.py`
- [Fault controls](https://github.com/hithaishisurendra/computer-use-automation/blob/main/coreserv/faults.py): `coreserv/faults.py`

### Discovery

- [Discovery entry](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/run.py): `discovery/run.py`
- [Observe-decide-act loop](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/loop.py): `discovery/loop.py`
- [Model providers](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/model.py): `discovery/model.py`
- [Discovery prompts](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/prompts.py): `discovery/prompts.py`
- [Accessibility perception](https://github.com/hithaishisurendra/computer-use-automation/blob/main/perception/tree.py): `perception/tree.py`
- [Label augmentation](https://github.com/hithaishisurendra/computer-use-automation/blob/main/perception/labeling.py): `perception/labeling.py`
- [Capability recorder](https://github.com/hithaishisurendra/computer-use-automation/blob/main/discovery/recorder.py): `discovery/recorder.py`

### Capability contract

- [Capability artifacts](https://github.com/hithaishisurendra/computer-use-automation/tree/main/capabilities): `capabilities`
- [Artifact schema](https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/schema.py): `capability/schema.py`
- [Artifact and overlay loader](https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/loader.py): `capability/loader.py`
- [Input and auth validation](https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/validate.py): `capability/validate.py`
- [Sensitive data redaction](https://github.com/hithaishisurendra/computer-use-automation/blob/main/capability/redaction.py): `capability/redaction.py`

### Deterministic replay

- [Replay entry](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/run.py): `replay/run.py`
- [Replay orchestration](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/engine.py): `replay/engine.py`
- [Browser action executor](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/executor.py): `replay/executor.py`
- [Element resolver](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/resolver.py): `replay/resolver.py`
- [Step checkpoints](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/checkpoints.py): `replay/checkpoints.py`
- [Outcome classification](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/classify.py): `replay/classify.py`
- [Replay result](https://github.com/hithaishisurendra/computer-use-automation/blob/main/replay/result.py): `replay/result.py`

### Human escalation

- [Session ownership](https://github.com/hithaishisurendra/computer-use-automation/blob/main/escalation/session.py): `escalation/session.py`
- [Intervention request](https://github.com/hithaishisurendra/computer-use-automation/blob/main/escalation/request.py): `escalation/request.py`
- [Handoff activity capture](https://github.com/hithaishisurendra/computer-use-automation/blob/main/escalation/capture.py): `escalation/capture.py`

### Other components

- **Calling operator**
- **Human operator**

## Connections

- Calling operator → Discovery entry: starts discovery (evidence: `README.md`)
- Calling operator → Replay entry: requests replay (evidence: `README.md`)
- Calling operator → CoreServ web app: uses console (evidence: `README.md`)
- CoreServ web app → Member data: reads member data (evidence: `coreserv/main.py`)
- CoreServ web app → Fault controls: checks fault flags (evidence: `coreserv/main.py`)
- Discovery entry → Observe-decide-act loop: starts loop (evidence: `discovery/run.py`)
- Discovery entry → Model providers: selects provider (evidence: `discovery/run.py`)
- Discovery entry → Capability recorder: creates recorder (evidence: `discovery/run.py`)
- Discovery entry → Artifact and overlay loader: resolves credentials (evidence: `discovery/run.py`)
- Observe-decide-act loop → Model providers: requests decisions (evidence: `discovery/loop.py`)
- Observe-decide-act loop → Discovery prompts: uses prompts (evidence: `discovery/prompts.py`)
- Observe-decide-act loop → Accessibility perception: observes page (evidence: `discovery/loop.py`)
- Accessibility perception → Label augmentation: augments labels (evidence: `perception/tree.py`)
- Observe-decide-act loop → Browser action executor: executes actions (evidence: `discovery/loop.py`)
- Observe-decide-act loop → Capability recorder: records run (evidence: `discovery/recorder.py`)
- Capability recorder → Artifact schema: builds artifact (evidence: `discovery/recorder.py`)
- Discovery entry → Capability artifacts: writes artifact (evidence: `README.md`)
- Capability artifacts → Replay entry: supplies capability (evidence: `README.md`)
- Replay entry → Artifact and overlay loader: loads capability (evidence: `replay/run.py`)
- Artifact and overlay loader → Artifact schema: parses artifact (evidence: `capability/loader.py`)
- Replay entry → Replay orchestration: runs replay (evidence: `replay/run.py`)
- Replay orchestration → Input and auth validation: validates inputs (evidence: `replay/engine.py`)
- Replay orchestration → Browser action executor: executes steps (evidence: `replay/engine.py`)
- Browser action executor → Element resolver: resolves elements (evidence: `replay/executor.py`)
- Replay orchestration → Accessibility perception: captures page state (evidence: `replay/engine.py`)
- Replay orchestration → Step checkpoints: evaluates checkpoints (evidence: `replay/engine.py`)
- Replay orchestration → Outcome classification: classifies outcome (evidence: `replay/engine.py`)
- Replay orchestration → Replay result: produces result (evidence: `replay/engine.py`)
- Replay orchestration → Session ownership: transfers control (evidence: `replay/engine.py`)
- Replay orchestration → Intervention request: requests intervention (evidence: `replay/engine.py`)
- Replay orchestration → Handoff activity capture: captures handoff (evidence: `replay/engine.py`)
- Observe-decide-act loop → Session ownership: supports takeover (evidence: `discovery/loop.py`)
- Observe-decide-act loop → Intervention request: requests intervention (evidence: `discovery/loop.py`)
- Human operator → Intervention request: receives request (evidence: `README.md`)
- Human operator → Session ownership: takes control (evidence: `README.md`)
- Replay entry → Sensitive data redaction: uses safe evidence (evidence: `replay/run.py`)
- Capability artifacts → Artifact and overlay loader: provides tenant overlay (evidence: `capability/loader.py`)