How artemisdicotiar/livecodebench works
Project type and purpose
• LiveCodeBench is a Python-based CLI benchmarking tool for holistically evaluating large-language-model code capabilities (generation, execution, self-repair, test output prediction) on continuously updated contest problems.
• It orchestrates dataset loading, prompt generation, model inference (via open-source or APIbased runners), and automated metric computation.
File structure
• Top-level:
• README.md, ERRATA.md, LICENSE, pyproject.toml, uv.lock
• assets/ (images for website/visualization)
• lcb_runner/ (main Python package)
• lcb_runner subpackages:
• benchmarks/ — definitions of codegen, execution, prediction scenarios
• prompts/ — prompt templates and few-shot examples, divided by scenario
• runner/ — CLI entrypoint (main.py), runner base class, per-model runners (oai_runner, vllm_runner, etc.), scenario router, parser for flags
• evaluation/ — metric calculators (pass@k), score aggregators, utilities for test execution
• utils/ — shared helper functions (path management, extraction, multiprocessing, scenario definitions)
• Config/deployment:
• pyproject.toml / uv.lock for dependency management
• lcb_sky.yml (Skyplane/DevOps config?)
Key insights from README
• Scenarios: code-generation, code-execution, test-output-prediction, self-repair
• Pluggable model framework via lm_styles.py and scenario_router
• Use of vllm for local GPU parallel inference, multiprocessing for closed-API models
• Outputs and evaluations are persisted for reuse/continuation
Main components & their interactions
a. CLI Layer
• Component: runner/main.py + parser.py
• Responsibility: parse flags, select scenario & model, hand off to ScenarioRouter
b. Scenario Router
• Decides between codegeneration, selfrepair, codeexecution, testoutputprediction
• Instantiates a Runner subclass
c. Runner Modules
• BaseRunner defines generic interface (load prompts, call model, dump outputs)
• Subclasses (oai_runner, vllm_runner, claude_runner, etc.) implement model‐specific API calls
• Handles caching, parallelization, timeouts
d. Prompt Manager
• Located under lcb_runner/prompts
• Templates + few-shot examples → scenario-specific prompt strings → fed to Runner
e. Model Inference (External Services)
• OpenAI, Anthropic, Cohere, vllm, etc.
• Network or local GPU calls
f. Persistence
• Output directory for generated code/results
• Intermediate caches for reuse
g. Evaluation Engine
• lcb_runner/evaluation computes pass@k, code execution success, test‐prediction accuracy
• compute_scores aggregates metrics over time windows
h. Reporting
• CLI prints metrics, writes score files
• Assets/images and website consume results for visualizations
Architectural patterns & design principles
• Modular plugin architecture: add new models by extending LMStyle & Runner subclasses
• Layered separation: CLI → Routing → Model-agnostic Runner → Model-specific implementation → Evaluation
• Reusability: prompts, utils, evaluation components can be shared across scenarios
• Extensibility: new scenarios or metrics can be dropped in their respective folders
Elements
[CLI Parser] → [ScenarioRouter] → [Runner: BaseRunner]
↳ [vllmRunner]
↳ [oaiRunner]
↳ …
[Prompt Manager]
[Evaluation Engine]
[Persistence Layer]
• User command → Parser → Router
• Router → Runner → Prompt Manager → Model Service → Runner → Output Store
• Output Store → Evaluation Engine → Score Aggregator
• Blue for internal modules (lcb_runner)
• Green for external dependencies (Model APIs, HF datasets)
• Yellow for data stores/files (cache, outputs, results)
• Rectangles for services/libraries
• Clouds for external services