How bigcode-project/bigcodebench works
Project Type and Purpose
• BigCodeBench is an open-source Python benchmarking toolkit and CLI for evaluating code generation capabilities of large language models (LLMs).
• It provides commands to generate code samples (bigcodebench.generate), remotely or locally execute those samples (bigcodebench.evaluate), sanitize outputs (bigcodebench.sanitize/syncheck), and aggregate/analyze results.
File Structure Analysis
• Top-level components:
• bigcodebench/ ← core Python package
• analysis/ ← post-evaluation scripts for result aggregation
• decontamination/ ← code-overlap checking modules and data
• Docker/ & sandbox-templates/ ← container and sandbox configs
• tests/ ← unit tests for sanitizers
• tools/ ← migration scripts for prior versions
• CLI wrappers: run.sh, release.sh
• Within bigcodebench/:
• gen/util/ ← provider adapters for OpenAI, Anthropic, Google, HF, Mistral, vLLM, E2B
• eval/ ← evaluation logic, oracles, utilities
• data/ ← dataset loader and utilities
• sanitize.py / syncheck.py ← sanitization pipelines
• inspect.py ← helper to inspect provider responses
Key README Insights
• Multi-backend support (vLLM, OpenAI, Anthropic, Google, HF, E2B) via adapter modules
• Remote evaluation via a Hugging Face Gradio space or an E2B sandbox
• Docker images specialized for generation, evaluation, and Gradio UI
• Result files and pass@k statistics saved locally