# artemisdicotiar/livecodebench architecture

> Architecture diagram of the GitHub repository artemisdicotiar/livecodebench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/artemisdicotiar/livecodebench
- Repository: https://github.com/artemisdicotiar/livecodebench

## Overview

##### Project type and purpose
- LiveCodeBench is a Python-based CLI benchmarking tool for holistically evaluating large-language-model code capabilities (generation, execution, self-repair, test output prediction) on continuously updated contest problems.
- It orchestrates dataset loading, prompt generation, model inference (via open-source or API­based runners), and automated metric computation.

##### File structure
- Top-level:
- README.md, ERRATA.md, LICENSE, pyproject.toml, uv.lock
- assets/ (images for website/visualization)
- lcb_runner/ (main Python package)
- lcb_runner subpackages:
- benchmarks/ — definitions of codegen, execution, prediction scenarios
- prompts/ — prompt templates and few-shot examples, divided by scenario
- runner/ — CLI entrypoint (main.py), runner base class, per-model runners (oai_runner, vllm_runner, etc.), scenario router, parser for flags
- evaluation/ — metric calculators (pass@k), score aggregators, utilities for test execution
- utils/ — shared helper functions (path management, extraction, multiprocessing, scenario definitions)
- Config/deployment:
- pyproject.toml / uv.lock for dependency management
- lcb_sky.yml (Skyplane/DevOps config?)

##### Key insights from README
- Scenarios: code-generation, code-execution, test-output-prediction, self-repair
- Pluggable model framework via lm_styles.py and scenario_router
- Use of vllm for local GPU parallel inference, multiprocessing for closed-API models
- Outputs and evaluations are persisted for reuse/continuation

##### Main components & their interactions
a. CLI Layer
- Component: runner/main.py + parser.py
- Responsibility: parse flags, select scenario & model, hand off to ScenarioRouter
b. Scenario Router
- Decides between codegeneration, selfrepair, codeexecution, testoutputprediction
- Instantiates a Runner subclass
c. Runner Modules
- BaseRunner defines generic interface (load prompts, call model, dump outputs)
- Subclasses (oai_runner, vllm_runner, claude_runner, etc.) implement model‐specific API calls
- Handles caching, parallelization, timeouts
d. Prompt Manager
- Located under lcb_runner/prompts
- Templates + few-shot examples → scenario-specific prompt strings → fed to Runner
e. Model Inference (External Services)
- OpenAI, Anthropic, Cohere, vllm, etc.
- Network or local GPU calls
f. Persistence
- Output directory for generated code/results
- Intermediate caches for reuse
g. Evaluation Engine
- lcb_runner/evaluation computes pass@k, code execution success, test‐prediction accuracy
- compute_scores aggregates metrics over time windows
h. Reporting
- CLI prints metrics, writes score files
- Assets/images and website consume results for visualizations

##### Architectural patterns & design principles
- Modular plugin architecture: add new models by extending LMStyle & Runner subclasses
- Layered separation: CLI → Routing → Model-agnostic Runner → Model-specific implementation → Evaluation
- Reusability: prompts, utils, evaluation components can be shared across scenarios
- Extensibility: new scenarios or metrics can be dropped in their respective folders

##### Elements
[CLI Parser] → [ScenarioRouter] → [Runner: BaseRunner]
↳ [vllmRunner]
↳ [oaiRunner]
↳ …
[Prompt Manager]
[Evaluation Engine]
[Persistence Layer]
- User command → Parser → Router
- Router → Runner → Prompt Manager → Model Service → Runner → Output Store
- Output Store → Evaluation Engine → Score Aggregator
- Blue for internal modules (lcb_runner)
- Green for external dependencies (Model APIs, HF datasets)
- Yellow for data stores/files (cache, outputs, results)
- Rectangles for services/libraries
- Clouds for external services

## Diagram (Mermaid)

```mermaid
graph TD
    %% Swimlane: User & CLI
    subgraph "User & CLI"
        direction TB
        User["User"]:::external
        Main["main.py\n(CLI Entrypoint)"]:::internal
        Parser["parser.py\n(Flag Parser)"]:::internal
    end

    %% Swimlane: Orchestration & Routing
    subgraph "Orchestration & Routing"
        direction TB
        Router["scenario_router.py\n(Scenario Router)"]:::internal
        BaseRunner["base_runner.py\n(BaseRunner Interface)"]:::internal
        subgraph "Prompt Manager"
            direction TB
            CGPrompt["code_generation.py"]:::internal
            CEPrompt["code_execution.py"]:::internal
            SRPrompt["self_repair.py"]:::internal
            TPPrompt["test_output_prediction.py"]:::internal
            FewShot["few_shot_examples/"]:::internal
        end
    end

    %% Swimlane: Inference Plugins
    subgraph "Inference Plugins"
        direction TB
        OAI["oai_runner.py\n(OpenAI)"]:::internal
        VLLM["vllm_runner.py\n(vLLM)"]:::internal
        Claude["claude_runner.py\n(Claude)"]:::internal
        Claude3["claude3_runner.py\n(Claude 3)"]:::internal
        Gemini["gemini_runner.py\n(Gemini)"]:::internal
        Mistral["mistral_runner.py\n(Mistral)"]:::internal
        Cohere["cohere_runner.py\n(Cohere)"]:::internal
        Deepseek["deepseek_runner.py\n(Deepseek)"]:::internal
        Fireworks["fireworks_runner.py\n(Fireworks)"]:::internal
        CustomEvalHook["custom_evaluator.py\n(Custom Hook)"]:::internal
    end

    %% Swimlane: External Services
    subgraph "External Services"
        direction TB
        OpenAIAPI["OpenAI"]:::external
        AnthropicAPI["Anthropic"]:::external
        CohereAPI["Cohere"]:::external
        HFData["HuggingFace Datasets"]:::external
        VLLMLocal["Local GPU (vLLM)"]:::external
    end

    %% Swimlane: Evaluation & Reporting
    subgraph "Evaluation & Reporting"
        direction TB
        EvalEngine["Evaluation Engine"]:::internal
        ScoreAgg["compute_scores.py\n(Score Aggregator)"]:::internal
        Report["Reporting & Assets"]:::internal
        OutputStore["Output/Cache\n(results files)"]:::storage
    end

    %% Connections
    User --> Main
    Main --> Parser
    Main --> Router
    Parser --> Router
    Router --> BaseRunner
    BaseRunner -->|loads prompts| CGPrompt
    BaseRunner -->|loads prompts| CEPrompt
    BaseRunner -->|loads prompts| SRPrompt
    BaseRunner -->|loads prompts| TPPrompt
    BaseRunner --> FewShot
    BaseRunner --> OAI
    BaseRunner --> VLLM
    BaseRunner --> Claude
    BaseRunner --> Claude3
    BaseRunner --> Gemini
    BaseRunner --> Mistral
    BaseRunner --> Cohere
    BaseRunner --> Deepseek
    BaseRunner --> Fireworks
    BaseRunner --> CustomEvalHook

    OAI -->|API call| OpenAIAPI
    Claude -->|API call| AnthropicAPI
    Cohere -->|API call| CohereAPI
    VLLM -->|local inference| VLLMLocal
    BaseRunner -->|loads datasets| HFData

    BaseRunner -->|writes outputs| OutputStore
    OutputStore --> EvalEngine
    EvalEngine --> ScoreAgg
    ScoreAgg --> Report
    Report --> OutputStore

    %% Click Events
    click Main "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/main.py"
    click Parser "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/parser.py"
    click Router "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/scenario_router.py"
    click BaseRunner "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/base_runner.py"
    click OAI "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/oai_runner.py"
    click VLLM "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/vllm_runner.py"
    click Claude "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/claude_runner.py"
    click Claude3 "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/claude3_runner.py"
    click Gemini "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/gemini_runner.py"
    click Mistral "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/mistral_runner.py"
    click Cohere "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/cohere_runner.py"
    click Deepseek "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/deepseek_runner.py"
    click Fireworks "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/fireworks_runner.py"
    click CustomEvalHook "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/runner/custom_evaluator.py"
    click CGPrompt "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/prompts/code_generation.py"
    click CEPrompt "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/prompts/code_execution.py"
    click SRPrompt "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/prompts/self_repair.py"
    click TPPrompt "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/prompts/test_output_prediction.py"
    click FewShot "https://github.com/artemisdicotiar/livecodebench/tree/main/lcb_runner/prompts/few_shot_examples/"
    click ScoreAgg "https://github.com/artemisdicotiar/livecodebench/blob/main/lcb_runner/evaluation/compute_scores.py"

    %% Styles
    classDef internal fill:#B3DDF2,stroke:#333,stroke-width:1px
    classDef external fill:#B2F2BB,stroke:#333,stroke-width:1px
    classDef storage fill:#FFE599,stroke:#333,stroke-width:1px
```
