# guosyjlu/ds-agent architecture

> Architecture diagram of the GitHub repository guosyjlu/ds-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/guosyjlu/ds-agent
- Repository: https://github.com/guosyjlu/ds-agent

## Overview

##### Project Overview and Core Purpose
- DS-Agent is built to automate data science tasks by using LLMs (e.g., GPT-3.5, GPT-4, and mixtral via vllm for deployment experiments) and case-based reasoning.
- The project is research-oriented (ICML 2024 paper) and provides a benchmark consisting of 30 data science tasks.
- It leverages modular agent implementations (agents/agent_autogpt.py, agent_langchain.py, etc.) and separates the experimentation workflow into a “development” path (for research runs and a case-based evaluation) and a “deployment” pipeline (for production code generation and evaluation).

##### Analyzing the File Structure – Key Components
- Root Level:
– README.md and an Excel file with raw development results.
- Deployment Directory:
– Contains scripts and configurations (e.g., code_generation.sh, code_evaluation.sh, start_api.sh) which handle code generation and evaluation of tasks.
– Holds configuration files (in deployment/config) that define settings for similarity ranking and LLM configurations.
– Includes various “experience_replay” scripts (e.g., airline-reviews.py, textual-entailment.py) that seem to cover use cases or specific task scenarios.
- Development Directory:
– Houses the MLAgentBench framework that facilitates the research experiments.
– Contains a package structure with modules under MLAgentBench including agent definitions (agent.py, agent_autogpt.py, etc.), environment configuration, and utilities (e.g., low_level_actions.py, high_level_actions.py, retrieval.py).
– Also includes setup files (setup.py, requirements.txt) and data assets (data.zip) to support experiments.

##### Main Components
a. Core Engine (Development Pipeline) – MLAgentBench:
- Modules and sub-modules:
- LLM Integration Module (LLM.py) where API keys for GPT-3.5/4 are set.
- Agent Modules (agents/agent.py, agent_autogpt.py, agent_langchain.py, agent_research.py, dsagent.py) representing different strategies.
- Environment and Action Modules (environment.py, high_level_actions.py, low_level_actions.py) for executing tasks.
- Experiment Orchestration (runner.py, prepare_task.py) that manage the flow of experiments and logging.
- Data Input Layer:
- Integration of benchmark datasets and Kaggle data. (Ensuring tasks in “development/benchmarks” and data in data.zip are represented.)

b. Deployment Pipeline:
- Generation and Evaluation Scripts:
- Code generation (generate.py, prompt.py) and evaluation (evaluation.py, execution.py) scripts.
- Experience Replay Scripts for different task scenarios (deployment/experience_replay/*).
- Configuration:
- Config files under deployment/config (similarity_ranking.json, heterogenous_similarity_ranking.json).
- LLM Server Integration:
- Scripts like start_api.sh to initialize an LLM server using the vllm framework for the production setting.
- Deployment’s LLM integration in generate.py where the open-source model (mixtral) can be set via configuration.

c. External Dependencies and Data Clients:
- External APIs for OpenAI LLMs as specified in both MLAgentBench/LLM.py and deployment/generate.py.
- vllm Framework for the open-sourced mixtral LLM.
- External benchmark data accessed via provided Google Drive links and preparation of custom datasets.

##### Relationships and Data Flow
- In the Development workflow, the experiment runner (runner.py) orchestrates data ingestion (from benchmark datasets and unzipped Kaggle data) and triggers various agent modules. The agents use LLM integration to generate data science solutions that are logged into “logs/” and “workspace/.”
- In the Deployment workflow, the generation scripts produce code or models based on a given prompt while evaluation scripts (with corresponding configurations) rank these outputs using the pre-defined metric. The experience replay directory contains task-specific experiments that may feed back insights to improve the agents.
- Data flow from benchmark datasets to the MLAgentBench experiment manager.
- Interaction between Agents and LLM modules.
- Flow between the deployment scripts (code generation → code evaluation) and the configuration modules.
- The initialization sequence starting with the LLM server via start_api.sh, which then interacts with the code generation process.

##### Architectural Patterns and Design Principles
- Modularity: Each component (agents, environment, evaluation, configuration) is separated into distinct modules.
- Separation of Concerns: Clear division between development (research and experimentation) and deployment (productionized experiments) pipelines.
- Extensibility: The agent modules are designed to extend or integrate different LLM approaches (e.g., AutoGPT, LangChain, etc.).
- Configuration-driven behavior: The presence of configuration files in deployment/config highlights an approach where behavior can be tuned without changing code.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data/Input and Dependencies
    data1["Benchmark Data (data.zip)"]:::data
    ext1["External LLM APIs"]:::ext

    %% Development Pipeline
    subgraph "Development Pipeline"
        direction TB
        d1["MLAgentBench Core Engine"]:::dev
        d2["LLM Integration Module"]:::dev
        d3["Agent AutoGPT"]:::dev
        d4["Agent LangChain"]:::dev
        d5["Agent Research"]:::dev
        d6["DS-Agent"]:::dev
        d7["Environment Setup"]:::dev
        d8["High-level Actions"]:::dev
        d9["Low-level Actions"]:::dev
        d10["Runner"]:::dev
        d11["Task Preparation"]:::dev
        d12["MLAgent Runner"]:::dev
    end

    %% Deployment Pipeline
    subgraph "Deployment Pipeline"
        direction TB
        de1["LLM Server Integration"]:::dep
        de2["Code Generation (Shell)"]:::dep
        de3["Generate Script"]:::dep
        de4["Code Evaluation (Shell)"]:::dep
        de5["Evaluation Script"]:::dep
        de6["Prompt Handler"]:::dep
        de7["Retrieval Support"]:::dep
        de8["Experience Replay"]:::dep
        de9["Deployment Config"]:::dep
    end

    %% Data Flow and Relationships
    %% Data/Input feeds into Development Runner
    data1 --> d10

    %% Development Pipeline internal flow
    d10 --> d1
    d1 --> d2
    d10 --> d3
    d10 --> d4
    d10 --> d5
    d10 --> d6
    d3 --> d2
    d4 --> d2
    d5 --> d2
    d6 --> d2
    d1 --> d7
    d7 --> d8
    d8 --> d9
    d10 --> d11
    d10 --> d12

    %% External LLM APIs interaction with development and deployment
    ext1 --> d2
    ext1 --> de3

    %% Deployment Pipeline Flow
    de1 --> de3
    de2 --> de3
    de3 --> de4
    de4 --> de5
    de6 --> de3
    de7 --> de3
    de9 --> de3
    de9 --> de5

    %% Feedback loop from Deployment to Development
    de8 --> d1

    %% Classes and Styles
    classDef dev fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef dep fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef data fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef ext fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;

    %% Click Events for Development Pipeline Components
    click d1 "https://github.com/guosyjlu/ds-agent/tree/main/development/MLAgentBench"
    click d2 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/LLM.py"
    click d3 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/agents/agent_autogpt.py"
    click d4 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/agents/agent_langchain.py"
    click d5 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/agents/agent_research.py"
    click d6 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/agents/dsagent.py"
    click d7 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/environment.py"
    click d8 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/high_level_actions.py"
    click d9 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/low_level_actions.py"
    click d10 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/runner.py"
    click d11 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/prepare_task.py"
    click d12 "https://github.com/guosyjlu/ds-agent/blob/main/development/MLAgentBench/mlagent_runner.py"
    click data1 "https://github.com/guosyjlu/ds-agent/blob/main/development/data.zip"

    %% Click Events for Deployment Pipeline Components
    click de2 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/code_generation.sh"
    click de3 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/generate.py"
    click de4 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/code_evaluation.sh"
    click de5 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/evaluation.py"
    click de6 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/prompt.py"
    click de7 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/retrieval.py"
    click de8 "https://github.com/guosyjlu/ds-agent/tree/main/deployment/experience_replay"
    click de9 "https://github.com/guosyjlu/ds-agent/tree/main/deployment/config"
    click de1 "https://github.com/guosyjlu/ds-agent/blob/main/deployment/start_api.sh"
```
