# sahityabiswas/scengen architecture

> Architecture diagram of the GitHub repository sahityabiswas/scengen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahityabiswas/scengen
- Repository: https://github.com/sahityabiswas/scengen

## Overview

#### Purpose and Entry Points
ScenGen is a Python 3.9 Android GUI-testing agent. Given an app ID and natural-language scenario, it repeatedly perceives the device UI, asks a local LLM for an action, executes it through ADB, and verifies progress. Primary entry point: `test.py`; launcher wrappers: `run_scengen_here.sh`, `run_scengen_here.bat`. `scen/` appears to be a parallel packaged copy of the core runtime.

#### Agent Orchestration and State
`TestAgent` owns the finite-state control loop and shared role coordination: initialize target app/scenario, observe, execute, loading/effect/end checks, then either continue, correct, fail, or finish.
- `testbot/core.py`
- `testbot/memory.py`
- `testbot/config.py`

App definitions and scenario definitions form the input boundary, resolved by IDs supplied to `test.py`:
- `testbot/apps.json`
- `testbot/scenarios.json`

#### Perception: Screenshot to UI Elements
The observer captures an Android screenshot and invokes UIED to produce a numbered, unified element list with text and bounding boxes. The detector retries transient blank screens. This output is the main boundary between visual perception and LLM reasoning.
- `testbot/roles/observer.py`
- `testbot/uied/detect.py`
- `testbot/uied/detect_merge/merge.py`

UIED runs two parallel branches: CV/connected-component discovery for visual regions and PaddleOCR recognition for text, then merges overlaps:
- `testbot/uied/detect_compo/ip_region_proposal.py`
- `testbot/uied/detect_text/ocr.py`
- `testbot/uied/detect_merge/merge.py`

PaddleOCR inference artifacts are committed in model directories, including `ch_PP-OCRv2_det_infer/` and `ch_PP-OCRv2_rec_infer/`.

#### Reasoning and Target Resolution
The action decider sends current annotated UI, scenario context, and action history to the LLM. It expects structured actions (`tap`, `input`, `scroll`, `back`, `wait`) and maps a textual widget target to detected elements through text matching and LLM-assisted fallback resolution.
- `testbot/roles/decider.py`
- `testbot/llm.py`
- `testbot/prompt/decider.py`

`LLMChatManager` separates decision and verification chat contexts; prompts are a contract boundary for model output parsing and behavior.

#### Device Execution and Verification
The executor converts resolved actions into ADB operations against a connected Android device. After each operation, the supervisor uses LLM visual checks to wait for loading completion, validate meaningful state change, determine scenario completion, and trigger correction retries.
- `testbot/roles/executor.py`
- `testbot/device.py`
- `testbot/roles/supervisor.py`

#### Runtime Integrations and Persistence
The runtime requires Python, ADB/device authorization, PaddlePaddle/PaddleOCR/OpenCV, and a locally served Ollama model (`llama3:8b`, expected at localhost:11434). No cloud LLM API is required. Recorder and logger persist execution evidence; runtime data directories include APK, bug, and log areas.
- `testbot/roles/recorder.py`
- `testbot/logger.py`
- `data/apk/`, `data/bug/`, `data/log/`

#### OCR Experimentation Boundary
`ocr_lab/` is an offline OCR evaluation/training workflow rather than the online agent loop: it manages manifests and ground truth, runs benchmarks, compares baseline/candidate predictions, and can train recognition models.
- `ocr_lab/scripts/run_pipeline.py`
- `ocr_lab/scripts/run_ocr_benchmark.py`
- `ocr_lab/scripts/train_rec_model.py`

Main entry points: [`testbot/core.py`](https://github.com/sahityabiswas/scengen/blob/main/testbot/core.py) (TestAgent control loop), [`testbot/roles/decider.py`](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/decider.py) (Action decider), [`testbot/llm.py`](https://github.com/sahityabiswas/scengen/blob/main/testbot/llm.py) (LLM chat manager), [`testbot/roles/supervisor.py`](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/supervisor.py) (Supervisor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Agent Runtime"]
  node_entry["CLI entry<br/>Python entry point<br/>[test.py]"]
  node_definitions["App &amp; scenario definitions<br/>JSON inputs<br/>[apps.json]"]
  node_agent{{"TestAgent control loop<br/>orchestrator<br/>[core.py]"}}
  node_memory["Shared agent state<br/>[memory.py]"]
  node_decider["Action decider<br/>role<br/>[decider.py]"]
  node_llm["LLM chat manager<br/>local LLM client<br/>[llm.py]"]
  node_prompts["Decision prompt contract<br/>[decider.py]"]
  node_executor["Executor<br/>role<br/>[executor.py]"]
  node_supervisor["Supervisor<br/>verification role<br/>[supervisor.py]"]
end

subgraph group_perception["UI Perception"]
  node_observer["Observer<br/>role<br/>[observer.py]"]
  node_uied["UIED detector<br/>perception pipeline<br/>[detect.py]"]
  node_cv["Visual region discovery<br/>CV branch"]
  node_ocr["PaddleOCR text recognition<br/>OCR branch<br/>[ocr.py]"]
  node_merge["Element merger<br/>fusion<br/>[merge.py]"]
end

subgraph group_integrations["Device, Model &amp; Evidence"]
  node_device["Android device / ADB<br/>device adapter<br/>[device.py]"]
  node_evidence[("Execution evidence<br/>recorder and logs<br/>[recorder.py]")]
  node_ollama{{"Local Ollama model<br/>LLM runtime"}}
end

subgraph group_ocr_lab["OCR Lab"]
  node_ocr_pipeline["OCR experiment pipeline<br/>offline workflow<br/>[run_pipeline.py]"]
  node_ocr_benchmark["OCR benchmark<br/>evaluation stage"]
  node_ocr_training["Recognition model training<br/>training stage<br/>[train_rec_model.py]"]
end

node_entry -->|"resolves IDs"| node_definitions
node_entry -->|"starts run"| node_agent
node_definitions -->|"target and scenario"| node_agent
node_agent -->|"reads and updates"| node_memory
node_agent -->|"observe"| node_observer
node_observer -->|"screenshot"| node_device
node_observer -->|"screenshots"| node_uied
node_uied -->|"parallel analysis"| node_cv
node_uied -->|"parallel analysis"| node_ocr
node_cv -->|"visual regions"| node_merge
node_ocr -->|"recognized text"| node_merge
node_merge -->|"annotated UI"| node_decider
node_agent -->|"decide next action"| node_decider
node_memory -->|"context and history"| node_decider
node_decider -->|"formats request"| node_prompts
node_decider -->|"decision request"| node_llm
node_prompts -->|"output contract"| node_llm
node_llm -->|"local inference"| node_ollama
node_decider -->|"resolved action"| node_executor
node_executor -->|"ADB operation"| node_device
node_agent -->|"verify progress"| node_supervisor
node_supervisor -->|"visual checks"| node_llm
node_supervisor -->|"continue, correct, fail, or finish"| node_agent
node_agent -->|"record run"| node_evidence
node_ocr_pipeline -->|"runs evaluation"| node_ocr_benchmark
node_ocr_pipeline -->|"runs training"| node_ocr_training

click node_entry "https://github.com/sahityabiswas/scengen/blob/main/test.py"
click node_definitions "https://github.com/sahityabiswas/scengen/blob/main/testbot/apps.json"
click node_agent "https://github.com/sahityabiswas/scengen/blob/main/testbot/core.py"
click node_memory "https://github.com/sahityabiswas/scengen/blob/main/testbot/memory.py"
click node_observer "https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/observer.py"
click node_uied "https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect.py"
click node_cv "https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect_compo/ip_region_proposal.py"
click node_ocr "https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect_text/ocr.py"
click node_merge "https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect_merge/merge.py"
click node_decider "https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/decider.py"
click node_llm "https://github.com/sahityabiswas/scengen/blob/main/testbot/llm.py"
click node_prompts "https://github.com/sahityabiswas/scengen/blob/main/testbot/prompt/decider.py"
click node_executor "https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/executor.py"
click node_device "https://github.com/sahityabiswas/scengen/blob/main/testbot/device.py"
click node_supervisor "https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/supervisor.py"
click node_evidence "https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/recorder.py"
click node_ocr_pipeline "https://github.com/sahityabiswas/scengen/blob/main/ocr_lab/scripts/run_pipeline.py"
click node_ocr_benchmark "https://github.com/sahityabiswas/scengen/blob/main/ocr_lab/scripts/run_ocr_benchmark.py"
click node_ocr_training "https://github.com/sahityabiswas/scengen/blob/main/ocr_lab/scripts/train_rec_model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_definitions,node_agent,node_memory,node_decider,node_llm,node_prompts,node_executor,node_supervisor toneBlue
class node_observer,node_uied,node_cv,node_ocr,node_merge toneAmber
class node_device,node_evidence,node_ollama toneMint
class node_ocr_pipeline,node_ocr_benchmark,node_ocr_training toneRose
```

## Components

### Agent Runtime

Online Android scenario execution loop.

- [CLI entry](https://github.com/sahityabiswas/scengen/blob/main/test.py): Python entry point · Starts a scenario run from app and scenario IDs. · `test.py`
- [App & scenario definitions](https://github.com/sahityabiswas/scengen/blob/main/testbot/apps.json): JSON inputs · Resolved into the target app and natural-language scenario. · `testbot/apps.json`
- [TestAgent control loop](https://github.com/sahityabiswas/scengen/blob/main/testbot/core.py): orchestrator · Coordinates observe, decide, execute, verify, retry, and finish states. · `testbot/core.py`
- [Shared agent state](https://github.com/sahityabiswas/scengen/blob/main/testbot/memory.py): state · Holds scenario context and action history. · `testbot/memory.py`
- [Action decider](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/decider.py): role · Selects structured actions and resolves widget targets. · `testbot/roles/decider.py`
- [LLM chat manager](https://github.com/sahityabiswas/scengen/blob/main/testbot/llm.py): local LLM client · Maintains separate decision and verification chat contexts. · `testbot/llm.py`
- [Decision prompt contract](https://github.com/sahityabiswas/scengen/blob/main/testbot/prompt/decider.py): prompt · Defines model output format and decision behavior. · `testbot/prompt/decider.py`
- [Executor](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/executor.py): role · Translates resolved actions into device operations. · `testbot/roles/executor.py`
- [Supervisor](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/supervisor.py): verification role · Checks loading, state change, completion, and correction retries. · `testbot/roles/supervisor.py`

### UI Perception

Screenshot analysis into actionable UI elements.

- [Observer](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/observer.py): role · Captures screens and retries transient blank views. · `testbot/roles/observer.py`
- [UIED detector](https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect.py): perception pipeline · Produces numbered UI elements with text and bounds. · `testbot/uied/detect.py`
- [Visual region discovery](https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect_compo/ip_region_proposal.py): CV branch · Finds connected-component visual regions. · `testbot/uied/detect_compo/ip_region_proposal.py`
- [PaddleOCR text recognition](https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect_text/ocr.py): OCR branch · Recognizes text in the screenshot. · `testbot/uied/detect_text/ocr.py`
- [Element merger](https://github.com/sahityabiswas/scengen/blob/main/testbot/uied/detect_merge/merge.py): fusion · Merges visual and text detections into unified elements. · `testbot/uied/detect_merge/merge.py`

### Device, Model & Evidence

Local runtime dependencies and persisted evidence.

- [Android device / ADB](https://github.com/sahityabiswas/scengen/blob/main/testbot/device.py): device adapter · Captures screenshots and executes Android interactions. · `testbot/device.py`
- [Execution evidence](https://github.com/sahityabiswas/scengen/blob/main/testbot/roles/recorder.py): recorder and logs · Persists run artifacts, bugs, and logs. · `testbot/roles/recorder.py`
- **Local Ollama model**: LLM runtime · Serves llama3:8b locally for decisions and visual verification.

### OCR Lab

Offline OCR evaluation and training workflow.

- [OCR experiment pipeline](https://github.com/sahityabiswas/scengen/blob/main/ocr_lab/scripts/run_pipeline.py): offline workflow · Manages OCR benchmarks, comparisons, and recognition-model training. · `ocr_lab/scripts/run_pipeline.py`
- [OCR benchmark](https://github.com/sahityabiswas/scengen/blob/main/ocr_lab/scripts/run_ocr_benchmark.py): evaluation stage · Evaluates baseline and candidate OCR predictions. · `ocr_lab/scripts/run_ocr_benchmark.py`
- [Recognition model training](https://github.com/sahityabiswas/scengen/blob/main/ocr_lab/scripts/train_rec_model.py): training stage · Trains OCR recognition models outside the agent loop. · `ocr_lab/scripts/train_rec_model.py`

## Connections

- CLI entry → App & scenario definitions: resolves IDs
- CLI entry → TestAgent control loop: starts run
- App & scenario definitions → TestAgent control loop: target and scenario
- TestAgent control loop → Shared agent state: reads and updates
- TestAgent control loop → Observer: observe
- Observer → Android device / ADB: screenshot
- Observer → UIED detector: screenshots
- UIED detector → Visual region discovery: parallel analysis
- UIED detector → PaddleOCR text recognition: parallel analysis
- Visual region discovery → Element merger: visual regions
- PaddleOCR text recognition → Element merger: recognized text
- Element merger → Action decider: annotated UI
- TestAgent control loop → Action decider: decide next action
- Shared agent state → Action decider: context and history
- Action decider → Decision prompt contract: formats request
- Action decider → LLM chat manager: decision request
- Decision prompt contract → LLM chat manager: output contract
- LLM chat manager → Local Ollama model: local inference
- Action decider → Executor: resolved action
- Executor → Android device / ADB: ADB operation
- TestAgent control loop → Supervisor: verify progress
- Supervisor → LLM chat manager: visual checks
- Supervisor → TestAgent control loop: continue, correct, fail, or finish
- TestAgent control loop → Execution evidence: record run
- OCR experiment pipeline → OCR benchmark: runs evaluation
- OCR experiment pipeline → Recognition model training: runs training