# hkuds/lightagent architecture

> Architecture diagram of the GitHub repository hkuds/lightagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hkuds/lightagent
- Repository: https://github.com/hkuds/lightagent

## Overview

##### Project Type and Purpose
- LightAgent is a full-stack research framework combining on-device and cloud LLMs to build mobile agents for GUI tasks.
- It consists of three major pillars:
a. Data Preparation Pipeline (prepare_data/)
b. Model Training Pipeline (model_training/)
c. Mobile Agent Deployment & Evaluation (agent/, evaluation/, vllm_script/, utils_mobile/, test_script/)

##### Key Architectural Components
A. On-Device Mobile Agent
- adb_client.py & utils_mobile/: Interfaces to control Android devices via ADB, capture screenshots.
- agent/: Core logic for task orchestration.
- agent/model.py – abstraction for LLM models
- agent/llm/ & agent/mllm/: wrappers for cloud & local LLMs (GLM, Qwen, Claude)
- agent/utils.py – task scheduling, memory mechanism
- page_executor/: Executors for interpreting screenshots (simple_vision_executor.py) and issuing text commands (text_executor.py).
- recorder/: Records interactions for debugging.

B. Cloud LLM Service
- vllm_script/run_server_visual_cot.py: Deploys vLLM server hosting larger cloud models.
- agent/llm/* & agent/mllm/*: Interfaces for remote calls (REST/gRPC) to cloud models.
- Configuration: configs/*.yaml specify model endpoints and thresholds.

C. Training & Data Pipelines
- prepare_data/: Scripts to generate synthetic vision-language chain-of-thought data and convert it into HuggingFace datasets.
- model_training/R1-V/: Two-stage training
- SFT: supervised fine-tuning using generated data.
- GRPO RL: reinforcement learning to optimize task completion.
- llama_factory_scripts/: Deepspeed configurations for large-scale training.

D. Evaluation & Benchmarking
- evaluation/: Wrapper around AndroidLab
- evaluation/config: YAML per mobile app task
- evaluation/tasks/: LLM-based evaluators replacing rule-based checks
- eval.py & generate_result.py: orchestrate tests and compile results
- test_script/: Batch run scripts for full and additional app suites
- logs/, app_data/: Store raw media and evaluation logs

##### Relationships & Data Flows
- User invokes eval.py or test_script → AndroidLab VM/emulator
- adb_client → captures screenshots → page_executor → extracts text/UI features
- agent orchestrator
- Step 1: call on-device model (agent/llm/glm4.py or local Qwen)
- Step 2: based on complexity/error, optionally escalate to cloud via vLLM server
- Responses → page_executor/text_executor → ADB controller → UI actions
- All interactions recorded in recorder/json_recoder.py → logs/
- Evaluation script uses LLM evaluators (evaluation/tasks/llm_evaluator.py) → generates metrics → generate_result.py

##### Architectural Patterns & Principles
- Layered Architecture:
- Device Control Layer (utils_mobile/, adb_client.py)
- Perception Layer (page_executor/)
- Reasoning Layer (agent/)
- Orchestration Layer (evaluation/)
- Client-Server Pattern: on-device agent as client, cloud LLM service as server
- Pipeline Pattern: data→preprocess→train→deploy→evaluate
- Separation of Concerns: distinct directories for data prep, training, runtime, evaluation

##### Technologies & Frameworks
- Python 3.x, ADB (Android Debug Bridge)
- Vision-Language Models: Qwen, GLM, Claude integration
- vLLM for efficient inference serving
- DeepSpeed & LLaMA-Factory for RL training
- AndroidLab benchmark for mobile GUI tasks
- YAML for configuration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Training Pipeline
    subgraph "Training Pipeline"
        style TrainingPipeline fill:#f96,stroke:#333,stroke-width:1px
        PrepareData["Data Preparation\n(prepare_data/)"]:::training
        ModelTraining["Model Training\n(model_training/R1-V/)"]:::training
        LLFactory["LLaMA-Factory Configs\n(model_training/llama_factory_scripts/)"]:::training
    end
    PrepareData --> ModelTraining --> LLFactory

    %% Runtime (On-Device Agent)
    subgraph "Runtime (On-Device Agent)"
        style Runtime fill:#9cf,stroke:#333,stroke-width:1px

        subgraph "Device Control Layer"
            ADBClient["ADB Client\n(adb_client.py)"]:::device
            UtilsMobile["Mobile Device Utilities\n(utils_mobile/)"]:::device
        end

        subgraph "Perception Layer"
            SimpleVision["Vision Executor\n(page_executor/simple_vision_executor.py)"]:::device
            TextExec["Text Executor\n(page_executor/text_executor.py)"]:::device
            ExecUtils["Executor Utils\n(page_executor/utils.py)"]:::device
        end

        subgraph "Reasoning Layer"
            AgentModel["Agent Model\n(agent/model.py)"]:::device
            AgentUtils["Agent Utils\n(agent/utils.py)"]:::device
        end

        subgraph "LLM Wrappers"
            GLM4["GLM4 Wrapper\n(agent/llm/glm4.py)"]:::cloud_llm
            QwenLLM["Qwen LLM Wrapper\n(agent/llm/qwen_llm_model.py)"]:::cloud_llm
            ClaudeMLLM["Claude Wrapper\n(agent/mllm/claude_model.py)"]:::cloud_llm
            QwenMLLM["Qwen MLLM Wrapper\n(agent/mllm/qwen_model.py)"]:::cloud_llm
        end

        Recorder["Interaction Recorder\n(recorder/json_recoder.py)"]:::device
    end

    %% Cloud LLM Service
    vLLMServer["vLLM Server\n(vllm_script/run_server_visual_cot.py)"]:::cloudservice

    %% Connect Runtime Flow
    ADBClient -->|screenshots| SimpleVision
    UtilsMobile -->|device I/O| SimpleVision
    SimpleVision --> ExecUtils --> AgentModel --> AgentUtils
    AgentUtils -->|local infer| GLM4
    AgentUtils -->|local infer| QwenLLM
    AgentUtils -->|local infer| ClaudeMLLM
    AgentUtils -->|local infer| QwenMLLM
    GLM4 -->|REST/gRPC| vLLMServer
    QwenLLM -->|REST/gRPC| vLLMServer
    ClaudeMLLM -->|REST/gRPC| vLLMServer
    QwenMLLM -->|REST/gRPC| vLLMServer
    vLLMServer -->|response| TextExec
    AgentUtils -->|chain-of-thought| Recorder
    SimpleVision -->|capture logs| Recorder
    TextExec -->|UI action| UtilsMobile
    TextExec -->|UI action| ADBClient

    %% Evaluation & Benchmarking
    subgraph "Evaluation & Benchmark"
        style Evaluation fill:#c9f,stroke:#333,stroke-width:1px
        EvalScript["Eval Orchestrator\n(eval.py)"]:::evaluation
        EvaluationDir["Evaluation Code\n(evaluation/)"]:::evaluation
        TestScripts["Batch Test Scripts\n(test_script/)"]:::evaluation
        GenerateResult["Result Generator\n(generate_result.py)"]:::evaluation
        LogsStore["Logs & App Data\n(logs/, app_data/)"]:::datastore
    end

    EvalScript -->|invokes| EvaluationDir
    EvaluationDir -->|runs| TestScripts
    TestScripts -->|stores| LogsStore
    EvaluationDir -->|writes| LogsStore
    EvaluationDir -->|calls| GenerateResult

    %% Configs
    Configs["Configuration Store\n(configs/)"]:::datastore
    Configs --> vLLMServer
    Configs --> AgentUtils
    Configs --> EvalScript

    %% Connections between Evaluation and Runtime
    EvalScript -->|runs agent| ADBClient

    %% Classes
    classDef training fill:#f96,stroke:#333;
    classDef device fill:#9cf,stroke:#333;
    classDef cloud_llm fill:#6f6,stroke:#333;
    classDef cloudservice fill:#6f6,stroke:#333,stroke-dasharray: 5 5;
    classDef evaluation fill:#c9f,stroke:#333;
    classDef datastore fill:#ccc,stroke:#333;

    %% Click Events
    click ADBClient "https://github.com/hkuds/lightagent/blob/main/adb_client.py"
    click UtilsMobile "https://github.com/hkuds/lightagent/tree/main/utils_mobile/"
    click SimpleVision "https://github.com/hkuds/lightagent/blob/main/page_executor/simple_vision_executor.py"
    click ExecUtils "https://github.com/hkuds/lightagent/blob/main/page_executor/utils.py"
    click TextExec "https://github.com/hkuds/lightagent/blob/main/page_executor/text_executor.py"
    click AgentModel "https://github.com/hkuds/lightagent/blob/main/agent/model.py"
    click AgentUtils "https://github.com/hkuds/lightagent/blob/main/agent/utils.py"
    click GLM4 "https://github.com/hkuds/lightagent/blob/main/agent/llm/glm4.py"
    click QwenLLM "https://github.com/hkuds/lightagent/blob/main/agent/llm/qwen_llm_model.py"
    click ClaudeMLLM "https://github.com/hkuds/lightagent/blob/main/agent/mllm/claude_model.py"
    click QwenMLLM "https://github.com/hkuds/lightagent/blob/main/agent/mllm/qwen_model.py"
    click Recorder "https://github.com/hkuds/lightagent/blob/main/recorder/json_recoder.py"
    click vLLMServer "https://github.com/hkuds/lightagent/blob/main/vllm_script/run_server_visual_cot.py"
    click PrepareData "https://github.com/hkuds/lightagent/tree/main/prepare_data/"
    click ModelTraining "https://github.com/hkuds/lightagent/tree/main/model_training/R1-V/"
    click LLFactory "https://github.com/hkuds/lightagent/tree/main/model_training/llama_factory_scripts/"
    click EvalScript "https://github.com/hkuds/lightagent/blob/main/eval.py"
    click EvaluationDir "https://github.com/hkuds/lightagent/tree/main/evaluation/"
    click TestScripts "https://github.com/hkuds/lightagent/tree/main/test_script/"
    click GenerateResult "https://github.com/hkuds/lightagent/blob/main/generate_result.py"
    click Configs "https://github.com/hkuds/lightagent/tree/main/configs/"
    click LogsStore "https://github.com/hkuds/lightagent/tree/main/logs/"
```
