How tencent-hunyuan/c3-benchmark works
Project type and purpose
• This is an open-source evaluation and data-generation framework for LLM-based agents (a combination of a benchmark suite and a controllable multi-agent data generation tool).
• Purpose:
• Generate high-quality multi-turn, multi-tool interaction tasks (multi_agent package)
• Run inference against various models and APIs, collect tool-call traces (bench_test/request_pipeline*.py)
• Results with fine-grained metrics (bench_test/analysis_result.py)
File structure
• Top-level:
• c3_bench/bench_test: evaluation pipelines, model handles, data inputs, results, analysis
• c3_bench/multi_agent: data generation framework (agents, planner, checker, utils)
• c3_bench/data: benchmark dataset
• root scripts: README, license, requirements.txt
• bench_test subdirectories:
• handle: model-specific wrappers for ToolCalls
• request_pipeline(.py, _upta.py): two pipeline variants
• result: output directory
• tool_class & tool_parser: definitions of tools and parsing logic
• analysis_result.py: post-run evaluation
• multi_agent subdirectories:
• agent: user, planner, tool, checker agent implementations
• tools: tool definitions and metadata
• utils: logging, file I/O, data processing
README for insights
• Two main workflows:
a. Inference & evaluation (bench_test)
b. Controllable multi-agent data generation (multi_agent)
• External dependencies: large language models via API or Hugging Face/vllm
• Emphasis on ToolCalls protocol for standardization
Main components
a. Data Layer
• C3-Bench dataset (JSONL/CSV)
• Generated raw/streamlined multi-agent data
b. Generation Engine (multi_agent)
• User agents, Planner agent, Tool agent, Checker agent
• Generation pipeline (generate.py)
c. Inference & Benchmark Runner (bench_test)
• Request pipelines (with and without UPTA)
• Model handles (APIHandle, Huggingface/vllm handles)
• ToolCalls orchestration (tool_class, tool_parser)
d. External Services
• LLM APIs (Hunyuan, OpenAI, etc.)
• vLLM/Hugging Face model servers
e. Evaluation & Analysis
• EvalByToolCallGraph module
• AnalysisResult module outputs CSV, metrics, visualization
Relationships and data flow
• Generation Engine → produces test tasks → writes to Data Layer
• Data Layer → fed into Request Pipelines → orchestrates ToolCalls → invokes External Services → collects raw responses → writes to bench_test/result
• bench_test/result → fed into AnalysisResult → outputs metrics and CSV files
• Throughout: ModelHandles and ToolParsers mediate between Request Pipelines and External Services
Architectural patterns and principles
• Modular separation: generation vs. benchmarking vs. analysis
• Plugin-style model handles: easily extend to new LLMs
• Standard ToolCalls protocol for all model interactions
• Pluggable tool definitions (tool_class + tool_parser)
• Separation of concerns: data, logic, I/O, analysis