How tencentqqgylab/appagent works
Project Type and Purpose
• AppAgent is a Python-based CLI tool/framework that turns large multimodal language models (e.g., GPT-4V or Qwen-VL) into agents capable of operating smartphone apps via ADB (Android Debug Bridge).
• It supports two phases: exploration (autonomous or human-guided) to build an app UI documentation knowledge base, and deployment to execute tasks using that knowledge.
a. CLI Entry Points
• learn.py (exploration mode)
• run.py (deployment mode)
b. Orchestration Layer (scripts/)
• self_explorer.py: autonomous UI exploration
• step_recorder.py: logs actions & screenshots
• document_generation.py: converts recorded steps into documentation artifacts
• task_executor.py: loads docs & drives the agent in deployment
• and_controller.py: issues ADB commands (screenshot, tap, swipe)
• model.py: wraps calls to GPT-4V or Qwen-VL API
• prompts.py: holds prompt templates for LLM queries
• utils.py & config.py: shared utilities, YAML config loader
c. Configuration & Assets
• config.yaml: API keys, model selection, request intervals
• assets/: stored documentation, test sets, license, teaser image
d. External Services
• OpenAI GPT-4 Vision or Alibaba Dashscope (Qwen-VL)
• Android device or emulator via ADB
Relationships & Data Flow
• User invokes learn.py or run.py → CLI parses config.yaml → loads orchestration modules
• Exploration Phase (learn.py):
• and_controller ↔ Device (ADB commands, screenshots)
• Screenshots + UI metadata → step_recorder → prompts + model
• LLM response → document_generation → assets/docs JSON files
• Deployment Phase (run.py):
• Load assets/docs → task_executor → prompts + model
• Model’s action plan → and_controller → Device
• step_recorder updates logs (optional)
• Throughout both phases, model.py abstracts API calls to external LLM service.
Architectural Patterns & Principles
• Modular layering:
• Presentation/CLI (learn.py, run.py)
• Orchestration/Business Logic (scripts/*.py)
• Service Abstraction (model wrapper, ADB controller)
• Persistence (local JSON docs in assets/)
• Separation of concerns: each script handles a single responsibility
• Extensibility point: adding new LLMs by extending scripts/model.py