How tencent-hunyuan/artifactsbenchmark works
Project Type and Purpose
• ArtifactsBench is an open-source evaluation toolkit for “visual-interactive” code generation with large language models (LLMs).
• It provides a CLI-driven pipeline to feed screenshot-based coding questions to remote (Gemini API) or local (Qwen2.5-VL via vLLM) multimodal LLM endpoints, then collects and scores their responses.
File‐Structure Overview
• Top‐level
• dataset/ — JSON questions (artifacts_bench.json)
• img/ — example result screenshots
• src/ — core Python modules
• test.sh — sample end-to-end script
• README.md, LICENSE
• src/
• code_parser.py — parses code from model replies
• extract_ans.py — post-processing answers for eval
• infer_gemini.py — orchestrates calls to remote Gemini API
• infer_qvl.py — orchestrates calls to local Qwen2.5-VL server
• prompts/ — prompt templates (e.g. prompt_mllm_check.py)
• utils.py — shared utilities (I/O, JSON, logging)
Key Components
a. Data Layer
• dataset/artifacts_bench.json (Input questions + model answers)
• screenshots_dir (PNG frames per question)
• save_path (results JSON with scores + reasons)
b. Inference Orchestrator
• infer_gemini.py (remote HTTP client using requests)
• infer_qvl.py (local vLLM OpenAI-compatible server client)
• concurrency (multiprocessing via --num_processes)
c. Prompt Engine
• src/prompts/prompt_mllm_check.py (builds multimodal prompt)
• tokenization (tokenizer_dir)
d. Model Endpoints
• Remote Gemini API (api_url + api_key)
e. Post-processing & Evaluation
• code_parser.py, extract_ans.py
• pytest-playwright (for any interactive checks)
f. Utilities
• src/utils.py (file I/O, JSON read/write, logging, argparse wrappers)
Relationships & Data Flow
• CLI (infer_gemini/infer_qvl) → Data Loader (reads dataset + screenshots)
→ Prompt Engine (injects screenshots, question into prompt)
→ Tokenizer (limits tokens)
→ Inference Client
• Gemini: HTTP(S) POST → Gemini Service → JSON response
• Qwen2.5: gRPC/HTTP to local vLLM server → JSON response
→ Code Parser & Extractor → Evaluation JSON → save_path
• test.sh wraps both pipelines for sanity checks.
Architectural Patterns & Principles
• Modular “Pipeline” pattern: clearly separated stages (load → prompt → infer → parse → save).
• Pluggable endpoints: swapping between Gemini and Qwen2.5 by changing scripts/parameters.
• Concurrency via multiprocessing pool.
• Separation of concerns: prompts, inference logic, parsing, utilities in distinct modules.
Technologies & Frameworks
• Python 3.x (+ vLLM, transformers, requests, tqdm)
• Playwright (for screenshot capture if needed, via pytest-playwright)
• JSON files as data interchange
• Multiprocessing for parallel inference