How greywarden07/touchbot works
Project type and purpose
• TouchBot is an agent-based automation framework (an open-source tool) for Android UI interaction and documentation.
• It supports two modes: human demonstration (records user taps/etc.) and autonomous exploration (agent explores automatically).
• It uses visual (screenshots) and XML UI dumps, feeds them to an LLM, then generates step-by-step documentation.
File structure
• Top-level scripts:
• learn.py & run.py: CLI entry points
• config.yaml: holds API keys & behavior settings
• requirements.txt: Python dependencies
• scripts/ directory: modular components
• and_controller.py: ADB/UI controller interface
• task_executor.py: executes UI actions (tap, swipe, etc.)
• step_recorder.py: captures human actions
• self_explorer.py: orchestrates autonomous exploration
• prompts.py: prompt templates
• model.py: LLM invocation layer
• document_generation.py: post-processing & doc creation
• utils.py & config.py: shared utilities & config loader
README insights
• Describes features, usage, modes, and where configuration lives.
• Mentions config.yaml for API keys (we will use Gemini API key).
Main components & interactions
a. Components
• CLI Interface (learn.py / run.py)
• Config Loader (scripts/config.py reading config.yaml)
• Mode Orchestrator
• HumanDemo Module (scripts/step_recorder.py)
• AutoExplore Module (scripts/self_explorer.py)
• UI Controller (scripts/and_controller.py)
• Task Executor (scripts/task_executor.py)
• Screenshot+XML Grabber (via and_controller.py)
• Prompt Templates (scripts/prompts.py)
• LLM Interface (scripts/model.py → Gemini API)
• Doc Generator (scripts/document_generation.py)
• Artifacts Store (tasks/ directory)
• External Service: Gemini API endpoint
b. Relationships
• CLI → reads config → invokes chosen mode
• Mode orchestrator → calls UI Controller & Task Executor → captures UI state
• Captured data → passed to Prompt Templates → packaged into LLM request
• Model layer → sends to Gemini API → receives JSON/text response
• Response → fed into Doc Generator → writes markdown/html into tasks/
c. Architectural patterns
• Modular pipeline: clear separation of data capture, reasoning, and output
• Adapter for external service (model.py abstracts Gemini calls)
• Strategy pattern for modes (HumanDemo vs AutoExplore)
d. Key technologies
• Python 3.8+
• ADB/UI Automator (via and_controller.py)
• Gemini LLM (instead of OpenAI)
• YAML config
Special note: Gemini API key
• Model.py reading “gemini_api_key” from config.yaml before making requests
Final advice
• Be as detailed as possible in showing each script’s responsibility.
• Essential flows without overcomplicating—group low-level utils under “Utilities.”
Layer 1 (top): CLI & Config
Layer 2: Mode Orchestrator
Layer 3: Capture & Execution
Layer 4: LLM Interface & External Service
Layer 5: Documentation Output