How thudm/cogagent works
Project Type & Purpose
• The project is an open-source GUI agent built around a vision-language model. Its purpose is to interpret GUI screenshots and textual prompt inputs to execute actions on a computer’s graphical interface.
• The README highlights its adoption in research and industrial contexts (e.g., CVPR 2024 highlight) and shows that it supports both inference and fine-tuning modalities (with separate pipelines for each).
Analyzing the File Structure
• Top-Level Directories:
• “app”: Contains the client-facing code including example scripts (client.py, openai_demo.py, register.py, vllm_openai_server.py), documentation, and requirement files for running the agent in a local environment.
• “finetune”: Contains the training configuration files (JSON and YAML configurations such as ds_zero_2.json, ds_zero_3.json, lora.yaml, sft.yaml) and the main finetuning script (finetune.py).
• “inference”: Contains demos for running the model in a command-line (cli_demo.py) or web-based (web_demo.py) interface.
• “assets”: Stores supporting media files (e.g., workflow images) used in the documentation.
• Miscellaneous files: Documentation (Action_space.md), LICENSE files, GitHub configuration files, etc.
• It is organized following a modular design where inference, application/demo usage, and fine-tuning are clearly separated—suggesting a design that segments core functionalities into distinct components.
Key Components & Their Roles
• Client/Application Component (App Directory):
• Acts as the front end to interact with users who send GUI operation requests.
• Contains scripts that form the “query” by combining tasks, history, and platform details.
• Inference Module (Inference Directory):
• Provides different interfaces (CLI and Web) to invoke the model and get responses.
• Fine-tuning Module (Finetune Directory):
• Includes configurations and scripts to fine-tune the model via either SFT or LoRA strategies.
• This module also documents GPU and memory requirements for training.
• Underlying VLM Model (CogAgent-9B-20241220):
• Based on GLM-4V-9B, it processes both images (sourced from GUI screenshots) and text inputs.
• It is responsible for generating GUI operations (like CLICK, TYPE, etc.) in response to user commands.
• External Dependencies:
• The project relies on specific versions of Python and GPU devices (e.g., A100/H100 or Ascend-based systems) as mentioned in the README.
• Although vLLM integration is mentioned, it is not supported yet; this is an external dependency to keep in mind for future expansion.
Relationships, Interactions & Architectural Patterns
• Data Flow:
• The user (or an external system) submits a task via the client interface (either through a CLI input or a web demo).
• The query is constructed using the provided input formats, combining tasks, historical operations, and platform identifiers.
• This query is passed into the inference module, which feeds it into the underlying VLM-based model.
• The model processes the input (using its trained GUI understanding) and produces a formatted output that includes action operations.
• Interaction Between Components:
• The App component interacts with both inference and external scripts as needed for a fully operational GUI agent.
• The fine-tuning pipeline is separate and typically used offline to update the model weights and configurations.
• Architectural Patterns:
• Modular Design – separation of concerns into “app,” “inference,” and “finetune” components.
• Pipeline Architecture – illustrating sequential stages from input preparation to inference and then output presentation.
• API Layer – especially in the web demo, where the model’s output is served as a REST-like response.