# open-compass/deveval architecture

> Architecture diagram of the GitHub repository open-compass/deveval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/open-compass/deveval
- Repository: https://github.com/open-compass/deveval

## Overview

##### Project Type and Purpose
- DevBench is a benchmarking and agent-driven automated software development framework. It orchestrates multiple phases of the software lifecycle—software design, environment setup, implementation, acceptance testing, and unit testing—by feeding curated code repositories into an LLM-based multi-agent system (a ChatDev derivative) and evaluating outputs automatically.

##### High-Level File Structure Analysis
- Top-level components:
- agent_system: The core orchestration engine (baseline ChatDev system)
- benchmark_data: A dataset of 22 curated sample projects across Python, C/C++, Java, JavaScript
- llm_judge: The LLM-as-judge evaluation module for software design outputs
- docker: Container definitions for reproducible environments
- agent_system substructure:
- baseline/camel: Core agents, message definitions, prompts, and model backend
- baseline/devagent: High-level phase composition, environment simulation, statistics
- baseline/online_log: A lightweight Flask (app.py) logging and visualization service
- CompanyConfig JSONs: Per-task, per-repo chat chain configurations
- run.py & test.py: Entry points for running the agent pipeline and quick tests
- llm_judge substructure:
- subeval/chat_api: Abstraction over LLM providers (OpenAI, local)
- subeval/subjective: Pairwise comparison algorithms, prompts, win-rate calculation
- scripts: Utilities to run evaluation jobs on design outputs

##### Key README Insights
- DevBench’s CLI drives agent_system with parameters: `--config` selects lifecycle phase, `--input_path` points to a sample repo, `--model` & `--model_source` configure the LLM, `--review` toggles peer code-review, `--evaluate` triggers post-run metrics (unit tests, acceptance tests, software design scoring via llm_judge).
- Dockerfiles define two images: base (model runtime) and complete (full DevBench with dependencies).

##### Components
a. Main Components
1. CLI Frontend: run.py entry point
2. Agent Orchestrator (agent_system):
– ChatChain Engine (devagent)
– Agent Modules (camel: chat_agent, task_agent, critic_agent, tool_agents)
– Model Backend (model_backend) connecting to LLM providers
– Phase Config Store (CompanyConfig JSONs)
3. LLM Providers:
– External OpenAI API
– Local Inference Servers (lmdeploy/vllm)
4. Logging & Visualization Service (online_log Flask app + static assets)
5. Benchmark Dataset Storage (benchmark_data)
6. Evaluation Module (llm_judge):
– Software Design Judge (pairwise comparison)
– Aggregation and metrics
7. Docker Infrastructure
b. Relationships and Data Flows
- User invokes CLI → Agent Orchestrator
- Orchestrator loads a target repo from benchmark_data or arbitrary input_path
- For each phase: Orchestrator reads CompanyConfig, spins up ChatChain
- ChatChain issues prompts → Model Backend → LLM Provider → returns completions
- Agents consume completions, possibly invoke tools (e.g., hugging_face_tool_agent)
- For Implementation & Testing phases: agents write code, run code locally → test results fed back to critic_agent if review=execution
- Orchestrator sends logs (messages, chain states, test outputs) to online_log service for real-time visualization
- Results are stored in output folders under llm_judge/output or printed to console
- Docker images encapsulate all above for reproducibility
c. Architectural Patterns and Principles
- Modular multi-agent pipeline (separation of concerns: roles, phases)
- Pluggable LLM backend (strategy pattern for different providers)
- Config-driven orchestration (ChatChainConfig JSON files per task)
- Micro-service for logging/visualization (Flask + static HTML/JS)
- Decoupled evaluation component (llm_judge) using pairwise LLM comparisons
d. Key Technologies and Frameworks
- Python 3 for core runtime
- Flask for online_log service
- JSON-based config for ChatChain parameters
- OpenAI API or HTTP endpoints for local models (via subeval/chat_api)
- Docker for containerization
- Standard testing frameworks for each language in benchmark_data

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Frontend
    CLI["CLI Frontend\n(run.py)"]:::core

    %% Agent System
    subgraph "Agent System"
        Orchestrator["ChatChain Orchestrator"]:::core
        ChatChainEngine["ChatChain Engine\n(Phase Composition & Env Simulation)"]:::core
        AgentModules["Agent Modules\n(chat_agent, task_agent, critic_agent, tool_agents)"]:::core
        ModelBackend["Model Backend\n(LLM Abstraction Layer)"]:::core
        Prompts["Prompts & Generators"]:::core
        Messages["Message Types & Utilities"]:::core
        PhaseConfig["Phase Config Store\n(CompanyConfig JSONs)"]:::datastore
        QuickTest["Quick Test Entry\n(test.py)"]:::core
    end

    %% External LLM Providers
    subgraph "LLM Providers" 
        OpenAI["OpenAI API"]:::external
        Local["Local Inference Server\n(lmdeploy/vllm)"]:::external
    end

    %% Logging Service
    LoggingService["Online Log Service\n(Flask + Static)"]:::micro

    %% Benchmark Data
    BenchmarkData["Benchmark Dataset\nStorage"]:::datastore

    %% Evaluation Module
    subgraph "Evaluation Module\n(llm_judge)"
        EvaluationModule["LLM-as-Judge Core"]:::core
        ChatAPI["LLM Provider Abstraction\n(chat_api)"]:::core
        Subjective["Pairwise Comparison\n(subjective)"]:::core
        EvalScripts["Evaluation Scripts & Utilities"]:::core
    end

    %% Docker Infrastructure
    subgraph "Docker Infrastructure"
        DockerBase["Base Image"]:::external
        DockerComplete["Complete Image"]:::external
    end

    %% Main Flows
    CLI -->|"invoke"| Orchestrator
    Orchestrator -->|loads repo from| BenchmarkData
    Orchestrator --> ChatChainEngine
    ChatChainEngine --> PhaseConfig
    ChatChainEngine --> AgentModules
    ChatChainEngine --> Prompts
    ChatChainEngine --> Messages
    ChatChainEngine --> ModelBackend
    ModelBackend --> OpenAI
    ModelBackend --> Local
    OpenAI --> ModelBackend
    Local --> ModelBackend
    ModelBackend --> ChatChainEngine
    ChatChainEngine --> Orchestrator
    Orchestrator --> LoggingService
    Orchestrator -->|on evaluate| EvaluationModule
    EvaluationModule --> ChatAPI
    EvaluationModule --> Subjective
    EvaluationModule --> EvalScripts
    DockerBase --> DockerComplete

    %% Click Events
    click CLI "https://github.com/open-compass/deveval/blob/main/agent_system/baseline/run.py"
    click Orchestrator "https://github.com/open-compass/deveval/blob/main/agent_system/baseline/devagent/chat_chain.py"
    click ChatChainEngine "https://github.com/open-compass/deveval/tree/main/agent_system/baseline/devagent/"
    click AgentModules "https://github.com/open-compass/deveval/tree/main/agent_system/baseline/camel/agents/"
    click ModelBackend "https://github.com/open-compass/deveval/blob/main/agent_system/baseline/camel/model_backend.py"
    click Prompts "https://github.com/open-compass/deveval/tree/main/agent_system/baseline/camel/prompts/"
    click Messages "https://github.com/open-compass/deveval/tree/main/agent_system/baseline/camel/messages/"
    click PhaseConfig "https://github.com/open-compass/deveval/tree/main/agent_system/baseline/CompanyConfig/"
    click QuickTest "https://github.com/open-compass/deveval/blob/main/agent_system/baseline/test.py"
    click LoggingService "https://github.com/open-compass/deveval/blob/main/agent_system/baseline/online_log/app.py"
    click BenchmarkData "https://github.com/open-compass/deveval/tree/main/benchmark_data/"
    click EvaluationModule "https://github.com/open-compass/deveval/tree/main/llm_judge/"
    click ChatAPI "https://github.com/open-compass/deveval/tree/main/llm_judge/subeval/chat_api/"
    click Subjective "https://github.com/open-compass/deveval/tree/main/llm_judge/subeval/subjective/"
    click EvalScripts "https://github.com/open-compass/deveval/tree/main/llm_judge/scripts/"
    click DockerBase "https://github.com/open-compass/deveval/tree/main/docker/base/dockerfile"
    click DockerComplete "https://github.com/open-compass/deveval/tree/main/docker/complete/dockerfile"

    %% Styles
    classDef core fill:#a1cfff,stroke:#333,stroke-width:1px
    classDef external fill:#c2f0c2,stroke:#333,stroke-width:1px
    classDef datastore fill:#ffd4a3,stroke:#333,stroke-width:1px
    classDef micro fill:#d3d3d3,stroke:#333,stroke-width:1px
```
