How guosyjlu/ds-agent works
Project Overview and Core Purpose
• DS-Agent is built to automate data science tasks by using LLMs (e.g., GPT-3.5, GPT-4, and mixtral via vllm for deployment experiments) and case-based reasoning.
• The project is research-oriented (ICML 2024 paper) and provides a benchmark consisting of 30 data science tasks.
• It leverages modular agent implementations (agents/agent_autogpt.py, agent_langchain.py, etc.) and separates the experimentation workflow into a “development” path (for research runs and a case-based evaluation) and a “deployment” pipeline (for production code generation and evaluation).
Analyzing the File Structure – Key Components
• Root Level:
– README.md and an Excel file with raw development results.
• Deployment Directory:
– Contains scripts and configurations (e.g., code_generation.sh, code_evaluation.sh, start_api.sh) which handle code generation and evaluation of tasks.
– Holds configuration files (in deployment/config) that define settings for similarity ranking and LLM configurations.
– Includes various “experience_replay” scripts (e.g., airline-reviews.py, textual-entailment.py) that seem to cover use cases or specific task scenarios.
• Development Directory:
– Houses the MLAgentBench framework that facilitates the research experiments.
– Contains a package structure with modules under MLAgentBench including agent definitions (agent.py, agent_autogpt.py, etc.), environment configuration, and utilities (e.g., low_level_actions.py, high_level_actions.py, retrieval.py).
– Also includes setup files (setup.py, requirements.txt) and data assets (data.zip) to support experiments.
Main Components
a. Core Engine (Development Pipeline) – MLAgentBench:
• Modules and sub-modules:
• LLM Integration Module (LLM.py) where API keys for GPT-3.5/4 are set.
• Agent Modules (agents/agent.py, agent_autogpt.py, agent_langchain.py, agent_research.py, dsagent.py) representing different strategies.
• Environment and Action Modules (environment.py, high_level_actions.py, low_level_actions.py) for executing tasks.
• Experiment Orchestration (runner.py, prepare_task.py) that manage the flow of experiments and logging.
• Data Input Layer:
• Integration of benchmark datasets and Kaggle data. (Ensuring tasks in “development/benchmarks” and data in data.zip are represented.)
b. Deployment Pipeline:
• Generation and Evaluation Scripts:
• Code generation (generate.py, prompt.py) and evaluation (evaluation.py, execution.py) scripts.
• Experience Replay Scripts for different task scenarios (deployment/experience_replay/*).
• Configuration:
• Config files under deployment/config (similarity_ranking.json, heterogenous_similarity_ranking.json).
• LLM Server Integration:
• Scripts like start_api.sh to initialize an LLM server using the vllm framework for the production setting.
• Deployment’s LLM integration in generate.py where the open-source model (mixtral) can be set via configuration.
c. External Dependencies and Data Clients:
• External APIs for OpenAI LLMs as specified in both MLAgentBench/LLM.py and deployment/generate.py.
• vllm Framework for the open-sourced mixtral LLM.
• External benchmark data accessed via provided Google Drive links and preparation of custom datasets.
Relationships and Data Flow
• In the Development workflow, the experiment runner (runner.py) orchestrates data ingestion (from benchmark datasets and unzipped Kaggle data) and triggers various agent modules. The agents use LLM integration to generate data science solutions that are logged into “logs/” and “workspace/.”
• In the Deployment workflow, the generation scripts produce code or models based on a given prompt while evaluation scripts (with corresponding configurations) rank these outputs using the pre-defined metric. The experience replay directory contains task-specific experiments that may feed back insights to improve the agents.
• Data flow from benchmark datasets to the MLAgentBench experiment manager.
• Interaction between Agents and LLM modules.
• Flow between the deployment scripts (code generation → code evaluation) and the configuration modules.
• The initialization sequence starting with the LLM server via start_api.sh, which then interacts with the code generation process.
Architectural Patterns and Design Principles
• Modularity: Each component (agents, environment, evaluation, configuration) is separated into distinct modules.
• Separation of Concerns: Clear division between development (research and experimentation) and deployment (productionized experiments) pipelines.
• Extensibility: The agent modules are designed to extend or integrate different LLM approaches (e.g., AutoGPT, LangChain, etc.).
• Configuration-driven behavior: The presence of configuration files in deployment/config highlights an approach where behavior can be tuned without changing code.