# tencent-hunyuan/artifactsbenchmark architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/artifactsbenchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/artifactsbenchmark
- Repository: https://github.com/tencent-hunyuan/artifactsbenchmark

## Overview

##### Project Type and Purpose
- ArtifactsBench is an open-source evaluation toolkit for “visual-interactive” code generation with large language models (LLMs).
- It provides a CLI-driven pipeline to feed screenshot-based coding questions to remote (Gemini API) or local (Qwen2.5-VL via vLLM) multimodal LLM endpoints, then collects and scores their responses.

##### File‐Structure Overview
- Top‐level
- dataset/ — JSON questions (artifacts_bench.json)
- img/ — example result screenshots
- src/ — core Python modules
- test.sh — sample end-to-end script
- README.md, LICENSE
- src/
- code_parser.py — parses code from model replies
- extract_ans.py — post-processing answers for eval
- infer_gemini.py — orchestrates calls to remote Gemini API
- infer_qvl.py — orchestrates calls to local Qwen2.5-VL server
- prompts/ — prompt templates (e.g. prompt_mllm_check.py)
- utils.py — shared utilities (I/O, JSON, logging)

##### Key Components
a. Data Layer
- dataset/artifacts_bench.json (Input questions + model answers)
- screenshots_dir (PNG frames per question)
- save_path (results JSON with scores + reasons)
b. Inference Orchestrator
- infer_gemini.py (remote HTTP client using requests)
- infer_qvl.py (local vLLM OpenAI-compatible server client)
- concurrency (multiprocessing via --num_processes)
c. Prompt Engine
- src/prompts/prompt_mllm_check.py (builds multimodal prompt)
- tokenization (tokenizer_dir)
d. Model Endpoints
- Remote Gemini API (api_url + api_key)
e. Post-processing & Evaluation
- code_parser.py, extract_ans.py
- pytest-playwright (for any interactive checks)
f. Utilities
- src/utils.py (file I/O, JSON read/write, logging, argparse wrappers)

##### Relationships & Data Flow
- CLI (infer_gemini/infer_qvl) → Data Loader (reads dataset + screenshots)
→ Prompt Engine (injects screenshots, question into prompt)
→ Tokenizer (limits tokens)
→ Inference Client
- Gemini: HTTP(S) POST → Gemini Service → JSON response
- Qwen2.5: gRPC/HTTP to local vLLM server → JSON response
→ Code Parser & Extractor → Evaluation JSON → save_path
- test.sh wraps both pipelines for sanity checks.

##### Architectural Patterns & Principles
- Modular “Pipeline” pattern: clearly separated stages (load → prompt → infer → parse → save).
- Pluggable endpoints: swapping between Gemini and Qwen2.5 by changing scripts/parameters.
- Concurrency via multiprocessing pool.
- Separation of concerns: prompts, inference logic, parsing, utilities in distinct modules.

##### Technologies & Frameworks
- Python 3.x (+ vLLM, transformers, requests, tqdm)
- Playwright (for screenshot capture if needed, via pytest-playwright)
- JSON files as data interchange
- Multiprocessing for parallel inference

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    dataset["artifacts_bench.json"]:::data
    screenshots["screenshots_dir (img/)"]:::data
    result["results JSON (save_path)"]:::data

    %% Pipeline Stages
    subgraph "Inference Pipeline"
        direction TB
        dataLoader["Data Loader"]:::module
        promptEngine["Prompt Engine"]:::module
        tokenizer["Tokenizer"]:::module
        subgraph "Inference Orchestrator"
            direction TB
            geminiScript["infer_gemini.py"]:::module
            qwenScript["infer_qvl.py"]:::module
        end
        subgraph "Model Endpoints"
            direction TB
            geminiAPI["Gemini API"]:::external
            vllmServer["Qwen2.5-VL Server"]:::external
        end
        codeParser["code_parser.py"]:::module
        extractor["extract_ans.py"]:::module
    end

    %% Utilities and CLI
    utils["utils.py"]:::module
    cli["test.sh / CLI"]:::module

    %% Data Flow
    dataset -->|"questions + answers"| dataLoader
    screenshots -->|"screenshots"| dataLoader
    dataLoader -->|"question + images"| promptEngine
    promptEngine -->|"prompt JSON"| tokenizer
    tokenizer -->|"token-limited prompt"| geminiScript
    tokenizer -->|"token-limited prompt"| qwenScript
    geminiScript -->|"HTTP POST"| geminiAPI
    qwenScript -->|"gRPC/HTTP"| vllmServer
    geminiAPI -->|"JSON response"| codeParser
    vllmServer -->|"JSON response"| codeParser
    codeParser -->|"parsed code"| extractor
    extractor -->|"scored results"| result

    cli --> dataLoader

    %% Utilities usage
    utils -.-> dataLoader
    utils -.-> promptEngine
    utils -.-> geminiScript
    utils -.-> qwenScript
    utils -.-> codeParser
    utils -.-> extractor

    %% Click Events
    click dataset "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/dataset/artifacts_bench.json"
    click screenshots "https://github.com/tencent-hunyuan/artifactsbenchmark/tree/main/img/"
    click geminiScript "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/src/infer_gemini.py"
    click qwenScript "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/src/infer_qvl.py"
    click promptEngine "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/src/prompts/prompt_mllm_check.py"
    click extractor "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/src/extract_ans.py"
    click codeParser "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/src/code_parser.py"
    click utils "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/src/utils.py"
    click cli "https://github.com/tencent-hunyuan/artifactsbenchmark/blob/main/test.sh"

    %% Styles
    classDef data fill:#ffeeba,stroke:#cc9a06,stroke-width:1px
    classDef module fill:#bee5eb,stroke:#31708f,stroke-width:1px
    classDef external fill:#d4edda,stroke:#155724,stroke-width:1px
```
