How sahityabiswas/scengen works
Purpose and Entry Points
ScenGen is a Python 3.9 Android GUI-testing agent. Given an app ID and natural-language scenario, it repeatedly perceives the device UI, asks a local LLM for an action, executes it through ADB, and verifies progress. Primary entry point: test.py; launcher wrappers: run_scengen_here.sh, run_scengen_here.bat. scen/ appears to be a parallel packaged copy of the core runtime.
Agent Orchestration and State
TestAgent owns the finite-state control loop and shared role coordination: initialize target app/scenario, observe, execute, loading/effect/end checks, then either continue, correct, fail, or finish.
• testbot/core.py
• testbot/memory.py
• testbot/config.py
App definitions and scenario definitions form the input boundary, resolved by IDs supplied to test.py:
• testbot/apps.json
• testbot/scenarios.json
Perception: Screenshot to UI Elements
The observer captures an Android screenshot and invokes UIED to produce a numbered, unified element list with text and bounding boxes. The detector retries transient blank screens. This output is the main boundary between visual perception and LLM reasoning.
• testbot/roles/observer.py
• testbot/uied/detect.py
• testbot/uied/detect_merge/merge.py
UIED runs two parallel branches: CV/connected-component discovery for visual regions and PaddleOCR recognition for text, then merges overlaps:
• testbot/uied/detect_compo/ip_region_proposal.py
• testbot/uied/detect_text/ocr.py
• testbot/uied/detect_merge/merge.py
PaddleOCR inference artifacts are committed in model directories, including ch_PP-OCRv2_det_infer/ and ch_PP-OCRv2_rec_infer/.
Reasoning and Target Resolution
The action decider sends current annotated UI, scenario context, and action history to the LLM. It expects structured actions (tap, input, scroll, back, wait) and maps a textual widget target to detected elements through text matching and LLM-assisted fallback resolution.
• testbot/roles/decider.py
• testbot/llm.py
• testbot/prompt/decider.py
LLMChatManager separates decision and verification chat contexts; prompts are a contract boundary for model output parsing and behavior.
Device Execution and Verification
The executor converts resolved actions into ADB operations against a connected Android device. After each operation, the supervisor uses LLM visual checks to wait for loading completion, validate meaningful state change, determine scenario completion, and trigger correction retries.
• testbot/roles/executor.py
• testbot/device.py
• testbot/roles/supervisor.py
Runtime Integrations and Persistence
The runtime requires Python, ADB/device authorization, PaddlePaddle/PaddleOCR/OpenCV, and a locally served Ollama model (llama3:8b, expected at localhost:11434). No cloud LLM API is required. Recorder and logger persist execution evidence; runtime data directories include APK, bug, and log areas.
• testbot/roles/recorder.py
• testbot/logger.py
• data/apk/, data/bug/, data/log/
OCR Experimentation Boundary
ocr_lab/ is an offline OCR evaluation/training workflow rather than the online agent loop: it manages manifests and ground truth, runs benchmarks, compares baseline/candidate predictions, and can train recognition models.
• ocr_lab/scripts/run_pipeline.py
• ocr_lab/scripts/run_ocr_benchmark.py
• ocr_lab/scripts/train_rec_model.py
Main entry points: core.py TestAgent control loop · decider.py Action decider · llm.py LLM chat manager · supervisor.py Supervisor
Components 20
Agent Runtime
Online Android scenario execution loop.
- CLI entry · Starts a scenario run from app and scenario IDs.
test.py - App & scenario definitions · Resolved into the target app and natural-language scenario.
testbot/apps.json - TestAgent control loop · Coordinates observe, decide, execute, verify, retry, and finish states.
testbot/core.py - Shared agent state · Holds scenario context and action history.
testbot/memory.py - Action decider · Selects structured actions and resolves widget targets.
testbot/roles/decider.py - LLM chat manager · Maintains separate decision and verification chat contexts.
testbot/llm.py - Decision prompt contract · Defines model output format and decision behavior.
testbot/prompt/decider.py - Executor · Translates resolved actions into device operations.
testbot/roles/executor.py - Supervisor · Checks loading, state change, completion, and correction retries.
testbot/roles/supervisor.py
UI Perception
Screenshot analysis into actionable UI elements.
- Observer · Captures screens and retries transient blank views.
testbot/roles/observer.py - UIED detector · Produces numbered UI elements with text and bounds.
testbot/uied/detect.py - Visual region discovery · Finds connected-component visual regions.
testbot/uied/detect_compo/ip_region_proposal.py - PaddleOCR text recognition · Recognizes text in the screenshot.
testbot/uied/detect_text/ocr.py - Element merger · Merges visual and text detections into unified elements.
testbot/uied/detect_merge/merge.py
Device, Model & Evidence
Local runtime dependencies and persisted evidence.
- Android device / ADB · Captures screenshots and executes Android interactions.
testbot/device.py - Execution evidence · Persists run artifacts, bugs, and logs.
testbot/roles/recorder.py - Local Ollama model · Serves llama3:8b locally for decisions and visual verification.
OCR Lab
Offline OCR evaluation and training workflow.
- OCR experiment pipeline · Manages OCR benchmarks, comparisons, and recognition-model training.
ocr_lab/scripts/run_pipeline.py - OCR benchmark · Evaluates baseline and candidate OCR predictions.
ocr_lab/scripts/run_ocr_benchmark.py - Recognition model training · Trains OCR recognition models outside the agent loop.
ocr_lab/scripts/train_rec_model.py
Connections 26
- CLI entry → App & scenario definitions: resolves IDs
- CLI entry → TestAgent control loop: starts run
- App & scenario definitions → TestAgent control loop: target and scenario
- TestAgent control loop → Shared agent state: reads and updates
- TestAgent control loop → Observer: observe
- Observer → Android device / ADB: screenshot
- Observer → UIED detector: screenshots
- UIED detector → Visual region discovery: parallel analysis
- UIED detector → PaddleOCR text recognition: parallel analysis
- Visual region discovery → Element merger: visual regions
- PaddleOCR text recognition → Element merger: recognized text
- Element merger → Action decider: annotated UI
- TestAgent control loop → Action decider: decide next action
- Shared agent state → Action decider: context and history
- Action decider → Decision prompt contract: formats request
- Action decider → LLM chat manager: decision request
- Decision prompt contract → LLM chat manager: output contract
- LLM chat manager → Local Ollama model: local inference
- Action decider → Executor: resolved action
- Executor → Android device / ADB: ADB operation
- TestAgent control loop → Supervisor: verify progress
- Supervisor → LLM chat manager: visual checks
- Supervisor → TestAgent control loop: continue, correct, fail, or finish
- TestAgent control loop → Execution evidence: record run
- OCR experiment pipeline → OCR benchmark: runs evaluation
- OCR experiment pipeline → Recognition model training: runs training