# tencent-hunyuan/c3-benchmark architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/c3-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/c3-benchmark
- Repository: https://github.com/tencent-hunyuan/c3-benchmark

## Overview

##### Project type and purpose
- This is an open-source evaluation and data-generation framework for LLM-based agents (a combination of a benchmark suite and a controllable multi-agent data generation tool).
- Purpose:
- Generate high-quality multi-turn, multi-tool interaction tasks (multi_agent package)
- Run inference against various models and APIs, collect tool-call traces (bench_test/request_pipeline*.py)
- Results with fine-grained metrics (bench_test/analysis_result.py)

##### File structure
- Top-level:
- c3_bench/bench_test: evaluation pipelines, model handles, data inputs, results, analysis
- c3_bench/multi_agent: data generation framework (agents, planner, checker, utils)
- c3_bench/data: benchmark dataset
- root scripts: README, license, requirements.txt
- bench_test subdirectories:
- handle: model-specific wrappers for ToolCalls
- request_pipeline(.py, _upta.py): two pipeline variants
- result: output directory
- tool_class & tool_parser: definitions of tools and parsing logic
- analysis_result.py: post-run evaluation
- multi_agent subdirectories:
- agent: user, planner, tool, checker agent implementations
- tools: tool definitions and metadata
- utils: logging, file I/O, data processing

##### README for insights
- Two main workflows:
a. Inference & evaluation (bench_test)
b. Controllable multi-agent data generation (multi_agent)
- External dependencies: large language models via API or Hugging Face/vllm
- Emphasis on ToolCalls protocol for standardization

##### Main components
a. Data Layer
- C3-Bench dataset (JSONL/CSV)
- Generated raw/streamlined multi-agent data
b. Generation Engine (multi_agent)
- User agents, Planner agent, Tool agent, Checker agent
- Generation pipeline (generate.py)
c. Inference & Benchmark Runner (bench_test)
- Request pipelines (with and without UPTA)
- Model handles (APIHandle, Huggingface/vllm handles)
- ToolCalls orchestration (tool_class, tool_parser)
d. External Services
- LLM APIs (Hunyuan, OpenAI, etc.)
- vLLM/Hugging Face model servers
e. Evaluation & Analysis
- EvalByToolCallGraph module
- AnalysisResult module outputs CSV, metrics, visualization

##### Relationships and data flow
- Generation Engine → produces test tasks → writes to Data Layer
- Data Layer → fed into Request Pipelines → orchestrates ToolCalls → invokes External Services → collects raw responses → writes to bench_test/result
- bench_test/result → fed into AnalysisResult → outputs metrics and CSV files
- Throughout: ModelHandles and ToolParsers mediate between Request Pipelines and External Services

##### Architectural patterns and principles
- Modular separation: generation vs. benchmarking vs. analysis
- Plugin-style model handles: easily extend to new LLMs
- Standard ToolCalls protocol for all model interactions
- Pluggable tool definitions (tool_class + tool_parser)
- Separation of concerns: data, logic, I/O, analysis

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Multi-Agent Data Generation
    subgraph "Multi-Agent Data Generation"
        Gen["Generation Engine\ngenerate.py"]:::code
        Agents["Agents\n(User, Planner, Tool, Checker)"]:::code
        Utils["Utils\n(logging, I/O, data)"]:::code
    end

    %% Data Storage
    subgraph "Data Storage"
        DatasetCSV["C3-Bench Dataset CSV"]:::data
        DatasetJSONL["C3-Bench Dataset JSONL"]:::data
        GenTasksEN["Generated Tasks EN"]:::data
        GenTasksZH["Generated Tasks ZH"]:::data
    end

    %% Benchmark Runner
    subgraph "Benchmark Runner"
        RP["Request Pipeline"]:::code
        RPU["Request Pipeline UPTA"]:::code
        Web["Web Interface"]:::code
        MH["Model Handles"]:::code
        TC["Tool Classes"]:::code
        TP["Tool Parsers"]:::code
        Results["Raw Results"]:::data
    end

    %% External Services
    subgraph "External Services"
        APIS["LLM APIs"]:::external
        vLLM["vLLM/HF Models"]:::external
    end

    %% Analysis & Reporting
    subgraph "Analysis & Reporting"
        TCG["Tool-Call Graph Evaluation"]:::code
        AR["Analysis Result"]:::code
    end

    %% Data flows for generation
    Gen -->|"JSONL I/O"| DatasetJSONL
    Gen -->|"JSONL I/O"| GenTasksEN
    Gen -->|"JSONL I/O"| GenTasksZH
    Agents --> Gen
    Utils --> Gen

    %% Generation to Benchmark
    DatasetJSONL -->|"JSONL I/O"| RP
    GenTasksEN --> RP
    GenTasksZH --> RP

    %% Benchmark pipelines
    RP -->|"ToolCalls"| MH
    RP --> TC
    RP --> TP
    RPU --> MH
    RPU --> TC
    RPU --> TP

    %% Benchmark to external
    MH -->|"HTTP/API"| APIS
    MH -->|"HTTP/API"| vLLM
    TC --> APIS
    TC --> vLLM
    TP --> APIS
    TP --> vLLM

    %% Results storage
    RP -->|"Raw JSON"| Results
    RPU --> Results
    Web --> RP
    Web --> RPU

    %% Analysis flow
    Results --> TCG
    Results --> AR
    TCG -->|"Metrics"| AR

    %% Click Events
    click Gen "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/multi_agent/generate.py"
    click Agents "https://github.com/tencent-hunyuan/c3-benchmark/tree/main/c3_bench/multi_agent/agent/"
    click Utils "https://github.com/tencent-hunyuan/c3-benchmark/tree/main/c3_bench/multi_agent/utils/"
    click DatasetCSV "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/data/C3-Bench.csv"
    click DatasetJSONL "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/data/C3-Bench.jsonl"
    click GenTasksEN "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/multi_agent/tools/tools_en.jsonl"
    click GenTasksZH "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/multi_agent/tools/tools_zh.jsonl"
    click RP "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/request_pipeline.py"
    click RPU "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/request_pipeline_upta.py"
    click Web "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/web_server.py"
    click MH "https://github.com/tencent-hunyuan/c3-benchmark/tree/main/c3_bench/bench_test/handle/"
    click TC "https://github.com/tencent-hunyuan/c3-benchmark/tree/main/c3_bench/bench_test/tool_class/"
    click TP "https://github.com/tencent-hunyuan/c3-benchmark/tree/main/c3_bench/bench_test/tool_parser/"
    click Results "https://github.com/tencent-hunyuan/c3-benchmark/tree/main/c3_bench/bench_test/result/"
    click AR "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/analysis_result.py"
    click TCG "https://github.com/tencent-hunyuan/c3-benchmark/blob/main/c3_bench/bench_test/tool_call_graph.py"

    %% Styles
    classDef code fill:#E0F7FA,stroke:#0288D1,color:#000
    classDef data fill:#E8F5E9,stroke:#388E3C,color:#000
    classDef external fill:#FFF3E0,stroke:#F57C00,color:#000
```
