# thudm/cogagent architecture

> Architecture diagram of the GitHub repository thudm/cogagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thudm/cogagent
- Repository: https://github.com/thudm/cogagent

## Overview

##### Project Type & Purpose
- The project is an open-source GUI agent built around a vision-language model. Its purpose is to interpret GUI screenshots and textual prompt inputs to execute actions on a computer’s graphical interface.
- The README highlights its adoption in research and industrial contexts (e.g., CVPR 2024 highlight) and shows that it supports both inference and fine-tuning modalities (with separate pipelines for each).

##### Analyzing the File Structure
- Top-Level Directories:
- “app”: Contains the client-facing code including example scripts (client.py, openai_demo.py, register.py, vllm_openai_server.py), documentation, and requirement files for running the agent in a local environment.
- “finetune”: Contains the training configuration files (JSON and YAML configurations such as ds_zero_2.json, ds_zero_3.json, lora.yaml, sft.yaml) and the main finetuning script (finetune.py).
- “inference”: Contains demos for running the model in a command-line (cli_demo.py) or web-based (web_demo.py) interface.
- “assets”: Stores supporting media files (e.g., workflow images) used in the documentation.
- Miscellaneous files: Documentation (Action_space.md), LICENSE files, GitHub configuration files, etc.
- It is organized following a modular design where inference, application/demo usage, and fine-tuning are clearly separated—suggesting a design that segments core functionalities into distinct components.

##### Key Components & Their Roles
- Client/Application Component (App Directory):
- Acts as the front end to interact with users who send GUI operation requests.
- Contains scripts that form the “query” by combining tasks, history, and platform details.
- Inference Module (Inference Directory):
- Provides different interfaces (CLI and Web) to invoke the model and get responses.
- Fine-tuning Module (Finetune Directory):
- Includes configurations and scripts to fine-tune the model via either SFT or LoRA strategies.
- This module also documents GPU and memory requirements for training.
- Underlying VLM Model (CogAgent-9B-20241220):
- Based on GLM-4V-9B, it processes both images (sourced from GUI screenshots) and text inputs.
- It is responsible for generating GUI operations (like CLICK, TYPE, etc.) in response to user commands.
- External Dependencies:
- The project relies on specific versions of Python and GPU devices (e.g., A100/H100 or Ascend-based systems) as mentioned in the README.
- Although vLLM integration is mentioned, it is not supported yet; this is an external dependency to keep in mind for future expansion.

##### Relationships, Interactions & Architectural Patterns
- Data Flow:
- The user (or an external system) submits a task via the client interface (either through a CLI input or a web demo).
- The query is constructed using the provided input formats, combining tasks, historical operations, and platform identifiers.
- This query is passed into the inference module, which feeds it into the underlying VLM-based model.
- The model processes the input (using its trained GUI understanding) and produces a formatted output that includes action operations.
- Interaction Between Components:
- The App component interacts with both inference and external scripts as needed for a fully operational GUI agent.
- The fine-tuning pipeline is separate and typically used offline to update the model weights and configurations.
- Architectural Patterns:
- Modular Design – separation of concerns into “app,” “inference,” and “finetune” components.
- Pipeline Architecture – illustrating sequential stages from input preparation to inference and then output presentation.
- API Layer – especially in the web demo, where the model’s output is served as a REST-like response.

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Interface / Client Application
    subgraph "User Interface / Client Application"
        UI["User Interface / Client Application"]:::client
    end

    %% Inference Engine with CLI and Web Demos
    subgraph "Inference Engine"
        CLI["CLI Demo"]:::inference
        WEB["Web Demo"]:::inference
    end

    %% Vision-Language Model Engine
    MODEL["Vision-Language Model Engine (GLM-4V-9B, CogAgent-9B-20241220)"]:::model

    %% Fine-tuning Pipeline
    subgraph "Fine-tuning Pipeline"
        FT["Fine-tuning Pipeline"]:::finetune
    end

    %% External Dependencies
    DEP["External Dependencies (GPU, Python, vLLM)"]:::external

    %% Data Flow Connections
    UI -->|"sends_query"| CLI
    UI -->|"sends_query"| WEB
    CLI -->|"forwards_query"| MODEL
    WEB -->|"forwards_query"| MODEL
    MODEL -->|"returns_GUI_Operations"| UI
    FT -.->|"updates_weights"| MODEL
    DEP ---|"supports"| MODEL

    %% Click Events for Client/Application Component
    click UI "https://github.com/thudm/cogagent/blob/main/app/client.py"
    click UI "https://github.com/thudm/cogagent/blob/main/app/openai_demo.py"
    click UI "https://github.com/thudm/cogagent/blob/main/app/register.py"
    click UI "https://github.com/thudm/cogagent/blob/main/app/vllm_openai_server.py"

    %% Click Events for Inference Module
    click CLI "https://github.com/thudm/cogagent/blob/main/inference/cli_demo.py"
    click WEB "https://github.com/thudm/cogagent/blob/main/inference/web_demo.py"

    %% Click Events for Fine-tuning Pipeline
    click FT "https://github.com/thudm/cogagent/blob/main/finetune/finetune.py"
    click FT "https://github.com/thudm/cogagent/tree/main/finetune/configs"

    %% Styles
    classDef client fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef inference fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef model fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef finetune fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef external fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
```
