# westlake-agi-lab/appagentx architecture

> Architecture diagram of the GitHub repository westlake-agi-lab/appagentx, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/westlake-agi-lab/appagentx
- Repository: https://github.com/westlake-agi-lab/appagentx

## Overview

##### Main Components
- Frontend/Demo Layer: Represented by the demo.py (and gradio-based demo) and the graphical interface provided by Gradio. This is the user interaction layer where the GUI agent’s results and execution are demonstrated.
- Backend Services: The backend directory contains two main modules:
- ImageEmbedding: Handles image-based feature extraction for screen recognition, wrapped in a Docker container.
- OmniParser: Focused on interface parsing utilizing LLM capabilities. It includes additional utility modules (e.g., action_matching.py, box_annotator.py) and weights management.
- LLM and Agent Logic: Various scripts (chain_evolve.py, chain_understand.py, explor_auto.py, explor_human.py) and the use of LangChain/LangGraph indicate the orchestration of the agent’s reasoning and evolution of high-level actions.
- Data/Memory Storage: The data folder includes components like State, graph_db.py, and vector_db.py that interact with Neo4j and Pinecone. This layer is critical for storing execution history and facilitating the memory mechanism.
- Tools and Utilities: Additional helper modules (in tool/ and utils.py) support image processing (img_tool) and screen content handling.
- Configuration & Deployment: Files like config.py, docker-compose.yml, Dockerfiles, and setup/requirements files guide deployment and external dependency configurations (e.g., ADB for Android, Docker for containerized services).

##### Relationships and Data Flow
- User Interaction starts with the Demo/Gradio interface, which triggers requests to the backend service.
- The backend, encompassing both ImageEmbedding and OmniParser modules, processes the screen information and parses the GUI to then feed data into the LLM-based reasoning system.
- The agent’s control logic (chain_evolve, chain_understand, etc.) directs actions, accesses previously stored execution data, and interacts with both the graph (Neo4j) and vector (Pinecone) databases.
- Tools and utilities provide supporting functions that are called by the main pipelines.
- Docker containers encapsulate the screen recognition and parsing tools, ensuring modularity and ease of deployment.

##### Architectural Patterns and Design Principles
- Modularization: The split into backend components (ImageEmbedding vs. OmniParser), data management modules, and frontends demonstrates clear separation of concerns.
- Containerization: With Dockerfiles and docker-compose, the system leverages containerization for easier deployment and scalability.
- Middleware/Integration Layer: The integration of external APIs and services (ADB, Neo4j, Pinecone, LangChain) shows the use of a middleware layer to manage data flows.
- Evolutionary Framework: The chain_evolve module suggests an evolutionary or iterative improvement pattern where the agent continuously refines its operations based on past execution data.

##### Relevant Technologies and Frameworks
- Frontend: Gradio for creating interactive demos.
- Backend: Python-based components using LangChain and LangGraph for LLM orchestration.
- Data Storage: Neo4j (graph DB) and Pinecone (vector DB).
- Containerization: Docker for environment isolation and GPU support when needed.
- External Dependencies: ADB for Android device communication; clearly noted configuration through config.py.

##### Visualization Elements
- Labels: Assign clear names to each module/component (e.g., “Screen Recognition”, “GUI Parsing”, “LLM Agent”, “Database - Neo4j”, “Database - Pinecone”).
- Annotations: Briefly annotate components with technology tags (e.g., “Docker”, “LangChain”, “ADB”) to provide clarity on the tools used.

##### Final Assembly
- On the right-hand side, structure a layered view:
- Top Layer: Agent Logic and orchestration (LLM, evolutionary chain modules).
- Middle Layer: Backend components (ImageEmbedding, OmniParser) possibly placed in separate containers.
- Bottom Layer: Data storage and external integrations (Neo4j, Pinecone, ADB).
- Auxiliary modules (Tools and Utilities) alongside the backend services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        F1["Frontend Demo"]:::frontend
        F2["Gradio Demo"]:::frontend
    end

    %% Backend Services
    subgraph "Backend Services"
        subgraph "Image Embedding Service"
            IE1["Image Embedding Code"]:::container
            IE2["Image Embedding Docker"]:::container
        end
        subgraph "OmniParser Service"
            OP1["OmniParser Core"]:::container
            OP2["OmniParser Parser"]:::container
            OP3["OmniParser Util"]:::container
        end
    end

    %% Agent Logic
    subgraph "Agent Logic"
        L1["Chain Evolve"]:::agent
        L2["Chain Understand"]:::agent
        L3["Explor Auto"]:::agent
        L4["Explor Human"]:::agent
    end

    %% Data/Memory Storage
    subgraph "Data/Memory Storage"
        D1["State"]:::database
        D2["Graph DB"]:::database
        D3["Vector DB"]:::database
        D4["Data Storage Helper"]:::database
    end

    %% Tools and Utilities
    subgraph "Tools and Utilities"
        T1["Img Tool"]:::util
        T2["Screen Content"]:::util
        T3["Utils"]:::util
    end

    %% Configuration & Deployment
    subgraph "Configuration & Deployment"
        C1["Config"]:::config
        C2["Docker Compose"]:::config
        C3["Other Dockerfiles"]:::config
    end

    %% Connections from Frontend to Backend
    F1 -->|"triggers"| IE1
    F1 -->|"triggers"| OP1
    F2 -->|"triggers"| OP1

    %% Connections from Backend Services to Agent Logic
    IE1 -->|"processes"| L1
    OP1 -->|"parses"| L1
    OP2 -->|"assists"| L1
    OP3 -->|"provides utils"| L1

    %% Interactions between Agent Logic and Tools/Utilities
    OP3 -->|"calls"| T1
    OP3 -->|"calls"| T2
    T3 -->|"supports"| L2

    %% Agent Logic to Data Storage
    L1 -->|"stores_history"| D1
    L2 -->|"queries"| D2
    L3 -->|"retrieves"| D3
    L4 -->|"updates"| D4

    %% Configuration influences Backend Services
    C1 -->|"configures"| IE2
    C1 -->|"configures"| OP3
    C2 -->|"orchestrates"| IE2
    C2 -->|"orchestrates"| OP2
    C3 -->|"deploys"| IE2
    C3 -->|"deploys"| OP2

    %% Styling
    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef container fill:#F4A460,stroke:#000,stroke-width:2px;
    classDef agent fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef database fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef util fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef config fill:#D3D3D3,stroke:#000,stroke-width:2px;

    %% Click Events
    click F1 "https://github.com/westlake-agi-lab/appagentx/blob/main/demo.py"
    click F2 "https://github.com/westlake-agi-lab/appagentx/blob/main/backend/OmniParser/gradio_demo.py"
    click IE1 "https://github.com/westlake-agi-lab/appagentx/blob/main/backend/ImageEmbedding/image_embedding.py"
    click IE2 "https://github.com/westlake-agi-lab/appagentx/tree/main/backend/ImageEmbedding/Dockerfile"
    click OP1 "https://github.com/westlake-agi-lab/appagentx/blob/main/backend/OmniParser/omni.py"
    click OP2 "https://github.com/westlake-agi-lab/appagentx/blob/main/backend/OmniParser/omniparser.py"
    click OP3 "https://github.com/westlake-agi-lab/appagentx/tree/main/backend/OmniParser/util/"
    click L1 "https://github.com/westlake-agi-lab/appagentx/blob/main/chain_evolve.py"
    click L2 "https://github.com/westlake-agi-lab/appagentx/blob/main/chain_understand.py"
    click L3 "https://github.com/westlake-agi-lab/appagentx/blob/main/explor_auto.py"
    click L4 "https://github.com/westlake-agi-lab/appagentx/blob/main/explor_human.py"
    click D1 "https://github.com/westlake-agi-lab/appagentx/blob/main/data/State.py"
    click D2 "https://github.com/westlake-agi-lab/appagentx/blob/main/data/graph_db.py"
    click D3 "https://github.com/westlake-agi-lab/appagentx/blob/main/data/vector_db.py"
    click D4 "https://github.com/westlake-agi-lab/appagentx/blob/main/data/data_storage.py"
    click T1 "https://github.com/westlake-agi-lab/appagentx/blob/main/tool/img_tool.py"
    click T2 "https://github.com/westlake-agi-lab/appagentx/blob/main/tool/screen_content.py"
    click T3 "https://github.com/westlake-agi-lab/appagentx/blob/main/utils.py"
    click C1 "https://github.com/westlake-agi-lab/appagentx/blob/main/config.py"
    click C2 "https://github.com/westlake-agi-lab/appagentx/blob/main/backend/docker-compose.yml"
    click C3 "https://github.com/westlake-agi-lab/appagentx/tree/main/backend/"
```
