How hkuds/lightagent works
Project Type and Purpose
• LightAgent is a full-stack research framework combining on-device and cloud LLMs to build mobile agents for GUI tasks.
• It consists of three major pillars:
a. Data Preparation Pipeline (prepare_data/)
b. Model Training Pipeline (model_training/)
c. Mobile Agent Deployment & Evaluation (agent/, evaluation/, vllm_script/, utils_mobile/, test_script/)
Key Architectural Components
A. On-Device Mobile Agent
• adb_client.py & utils_mobile/: Interfaces to control Android devices via ADB, capture screenshots.
• agent/: Core logic for task orchestration.
• agent/model.py – abstraction for LLM models
• agent/llm/ & agent/mllm/: wrappers for cloud & local LLMs (GLM, Qwen, Claude)
• agent/utils.py – task scheduling, memory mechanism
• page_executor/: Executors for interpreting screenshots (simple_vision_executor.py) and issuing text commands (text_executor.py).
• recorder/: Records interactions for debugging.
B. Cloud LLM Service
• vllm_script/run_server_visual_cot.py: Deploys vLLM server hosting larger cloud models.
• agent/llm/* & agent/mllm/*: Interfaces for remote calls (REST/gRPC) to cloud models.
• Configuration: configs/*.yaml specify model endpoints and thresholds.
C. Training & Data Pipelines
• prepare_data/: Scripts to generate synthetic vision-language chain-of-thought data and convert it into HuggingFace datasets.
• model_training/R1-V/: Two-stage training
• SFT: supervised fine-tuning using generated data.
• GRPO RL: reinforcement learning to optimize task completion.
• llama_factory_scripts/: Deepspeed configurations for large-scale training.
D. Evaluation & Benchmarking
• evaluation/: Wrapper around AndroidLab
• evaluation/config: YAML per mobile app task
• evaluation/tasks/: LLM-based evaluators replacing rule-based checks
• eval.py & generate_result.py: orchestrate tests and compile results
• test_script/: Batch run scripts for full and additional app suites
• logs/, app_data/: Store raw media and evaluation logs
Relationships & Data Flows
• User invokes eval.py or test_script → AndroidLab VM/emulator
• adb_client → captures screenshots → page_executor → extracts text/UI features
• agent orchestrator
• Step 1: call on-device model (agent/llm/glm4.py or local Qwen)
• Step 2: based on complexity/error, optionally escalate to cloud via vLLM server
• Responses → page_executor/text_executor → ADB controller → UI actions
• All interactions recorded in recorder/json_recoder.py → logs/
• Evaluation script uses LLM evaluators (evaluation/tasks/llm_evaluator.py) → generates metrics → generate_result.py
Architectural Patterns & Principles
• Layered Architecture:
• Device Control Layer (utils_mobile/, adb_client.py)
• Perception Layer (page_executor/)
• Reasoning Layer (agent/)
• Orchestration Layer (evaluation/)
• Client-Server Pattern: on-device agent as client, cloud LLM service as server
• Pipeline Pattern: data→preprocess→train→deploy→evaluate
• Separation of Concerns: distinct directories for data prep, training, runtime, evaluation
Technologies & Frameworks
• Python 3.x, ADB (Android Debug Bridge)
• Vision-Language Models: Qwen, GLM, Claude integration
• vLLM for efficient inference serving
• DeepSpeed & LLaMA-Factory for RL training
• AndroidLab benchmark for mobile GUI tasks
• YAML for configuration