How open-compass/deveval works
Project Type and Purpose
• DevBench is a benchmarking and agent-driven automated software development framework. It orchestrates multiple phases of the software lifecycle—software design, environment setup, implementation, acceptance testing, and unit testing—by feeding curated code repositories into an LLM-based multi-agent system (a ChatDev derivative) and evaluating outputs automatically.
High-Level File Structure Analysis
• Top-level components:
• agent_system: The core orchestration engine (baseline ChatDev system)
• benchmark_data: A dataset of 22 curated sample projects across Python, C/C++, Java, JavaScript
• llm_judge: The LLM-as-judge evaluation module for software design outputs
• docker: Container definitions for reproducible environments
• agent_system substructure:
• baseline/camel: Core agents, message definitions, prompts, and model backend
• baseline/devagent: High-level phase composition, environment simulation, statistics
• baseline/online_log: A lightweight Flask (app.py) logging and visualization service
• CompanyConfig JSONs: Per-task, per-repo chat chain configurations
• run.py & test.py: Entry points for running the agent pipeline and quick tests
• llm_judge substructure:
• subeval/chat_api: Abstraction over LLM providers (OpenAI, local)
• subeval/subjective: Pairwise comparison algorithms, prompts, win-rate calculation
• scripts: Utilities to run evaluation jobs on design outputs
Key README Insights
• DevBench’s CLI drives agent_system with parameters: --config selects lifecycle phase, --input_path points to a sample repo, --model & --model_source configure the LLM, --review toggles peer code-review, --evaluate triggers post-run metrics (unit tests, acceptance tests, software design scoring via llm_judge).
• Dockerfiles define two images: base (model runtime) and complete (full DevBench with dependencies).
Components
a. Main Components
1. CLI Frontend: run.py entry point
2. Agent Orchestrator (agent_system):
– ChatChain Engine (devagent)
– Agent Modules (camel: chat_agent, task_agent, critic_agent, tool_agents)
– Model Backend (model_backend) connecting to LLM providers
– Phase Config Store (CompanyConfig JSONs)
3. LLM Providers:
– External OpenAI API
– Local Inference Servers (lmdeploy/vllm)
4. Logging & Visualization Service (online_log Flask app + static assets)
5. Benchmark Dataset Storage (benchmark_data)
6. Evaluation Module (llm_judge):
– Software Design Judge (pairwise comparison)
– Aggregation and metrics
7. Docker Infrastructure
b. Relationships and Data Flows
• User invokes CLI → Agent Orchestrator
• Orchestrator loads a target repo from benchmark_data or arbitrary input_path
• For each phase: Orchestrator reads CompanyConfig, spins up ChatChain
• ChatChain issues prompts → Model Backend → LLM Provider → returns completions
• Agents consume completions, possibly invoke tools (e.g., hugging_face_tool_agent)
• For Implementation & Testing phases: agents write code, run code locally → test results fed back to critic_agent if review=execution
• Orchestrator sends logs (messages, chain states, test outputs) to online_log service for real-time visualization
• Results are stored in output folders under llm_judge/output or printed to console
• Docker images encapsulate all above for reproducibility
c. Architectural Patterns and Principles
• Modular multi-agent pipeline (separation of concerns: roles, phases)
• Pluggable LLM backend (strategy pattern for different providers)
• Config-driven orchestration (ChatChainConfig JSON files per task)
• Micro-service for logging/visualization (Flask + static HTML/JS)
• Decoupled evaluation component (llm_judge) using pairwise LLM comparisons
d. Key Technologies and Frameworks
• Python 3 for core runtime
• Flask for online_log service
• JSON-based config for ChatChain parameters
• OpenAI API or HTTP endpoints for local models (via subeval/chat_api)
• Docker for containerization
• Standard testing frameworks for each language in benchmark_data