How hithaishisurendra/computer-use-automation works
This repository automates a hostile back-office UI: an LLM-led discovery run records a typed capability, then a separate deterministic replay executes it without a model. The product also supports tenant overlays, outcome classification, and human takeover of a live session. The graph follows documented entry points and source-index references; several implementation details are unsampled, so uncertain links are omitted rather than inferred.
Main entry points: engine.py Replay orchestration · loop.py Observe-decide-act loop · discovery/run.py Discovery entry · replay/run.py Replay entry
Components 27
CoreServ target
- CoreServ web app
coreserv/main.py - Member data
coreserv/data.py - Fault controls
coreserv/faults.py
Discovery
- Discovery entry
discovery/run.py - Observe-decide-act loop
discovery/loop.py - Model providers
discovery/model.py - Discovery prompts
discovery/prompts.py - Accessibility perception
perception/tree.py - Label augmentation
perception/labeling.py - Capability recorder
discovery/recorder.py
Capability contract
- Capability artifacts
capabilities - Artifact schema
capability/schema.py - Artifact and overlay loader
capability/loader.py - Input and auth validation
capability/validate.py - Sensitive data redaction
capability/redaction.py
Deterministic replay
- Replay entry
replay/run.py - Replay orchestration
replay/engine.py - Browser action executor
replay/executor.py - Element resolver
replay/resolver.py - Step checkpoints
replay/checkpoints.py - Outcome classification
replay/classify.py - Replay result
replay/result.py
Human escalation
- Session ownership
escalation/session.py - Intervention request
escalation/request.py - Handoff activity capture
escalation/capture.py
Other components
- Calling operator
- Human operator
Connections 37
- Calling operator → Discovery entry: starts discovery
README.md - Calling operator → Replay entry: requests replay
README.md - Calling operator → CoreServ web app: uses console
README.md - CoreServ web app → Member data: reads member data
coreserv/main.py - CoreServ web app → Fault controls: checks fault flags
coreserv/main.py - Discovery entry → Observe-decide-act loop: starts loop
discovery/run.py - Discovery entry → Model providers: selects provider
discovery/run.py - Discovery entry → Capability recorder: creates recorder
discovery/run.py - Discovery entry → Artifact and overlay loader: resolves credentials
discovery/run.py - Observe-decide-act loop → Model providers: requests decisions
discovery/loop.py - Observe-decide-act loop → Discovery prompts: uses prompts
discovery/prompts.py - Observe-decide-act loop → Accessibility perception: observes page
discovery/loop.py - Accessibility perception → Label augmentation: augments labels
perception/tree.py - Observe-decide-act loop → Browser action executor: executes actions
discovery/loop.py - Observe-decide-act loop → Capability recorder: records run
discovery/recorder.py - Capability recorder → Artifact schema: builds artifact
discovery/recorder.py - Discovery entry → Capability artifacts: writes artifact
README.md - Capability artifacts → Replay entry: supplies capability
README.md - Replay entry → Artifact and overlay loader: loads capability
replay/run.py - Artifact and overlay loader → Artifact schema: parses artifact
capability/loader.py - Replay entry → Replay orchestration: runs replay
replay/run.py - Replay orchestration → Input and auth validation: validates inputs
replay/engine.py - Replay orchestration → Browser action executor: executes steps
replay/engine.py - Browser action executor → Element resolver: resolves elements
replay/executor.py - Replay orchestration → Accessibility perception: captures page state
replay/engine.py - Replay orchestration → Step checkpoints: evaluates checkpoints
replay/engine.py - Replay orchestration → Outcome classification: classifies outcome
replay/engine.py - Replay orchestration → Replay result: produces result
replay/engine.py - Replay orchestration → Session ownership: transfers control
replay/engine.py - Replay orchestration → Intervention request: requests intervention
replay/engine.py - Replay orchestration → Handoff activity capture: captures handoff
replay/engine.py - Observe-decide-act loop → Session ownership: supports takeover
discovery/loop.py - Observe-decide-act loop → Intervention request: requests intervention
discovery/loop.py - Human operator → Intervention request: receives request
README.md - Human operator → Session ownership: takes control
README.md - Replay entry → Sensitive data redaction: uses safe evidence
replay/run.py - Capability artifacts → Artifact and overlay loader: provides tenant overlay
capability/loader.py