How megh-zyke/efficiency-benchmark works
Project type and purpose
• This is a Python-based benchmarking framework that generates, executes, and scores LLM-written algorithmic code for efficiency (time + memory) and correctness.
• It drives multiple LLM providers (Hugging Face models, Groq, Gemini, etc.), collects their outputs, runs stress tests, computes unified scores, and produces CSV results + visualizations.
File structure
• Top-level scripts & config:
• Dockerfile / .dockerignore – containerizes the environment
• requirements.txt – Python dependencies
• README.md – project overview
• data/ – stores raw JSON outputs from various LLMs (e.g. gemini-2.5-pro.json)
• src/ – core pipeline code:
• openSourceLLMoutput.py & groqLLMoutput.py – fetch code from external LLM APIs
• compute.py & computeScore.py – run tests, measure time/memory, calculate scores
• imports.txt – shared import definitions
• packages/ – utility classes for constructing test inputs & parsing outputs (ListNode, TreeNode, etc.)
• results/ – stores per-model CSV results & a visualization script (visualisation.py)
README
• Confirms pipeline stages: LLM inference → data storage → compute metrics → results + plots
• Lists evaluation metrics (correctness, execution time, memory) and scoring formula
• Mentions stress-testing levels and dataset sources
Main components & their relationships
a. LLM Inference Layer
• Components: openSourceLLMoutput.py, groqLLMoutput.py, prompts/prompt.py
• External Services: Hugging Face API, Groq LLM endpoint, other LLM providers
• Output: JSON files in data/ directory
b. Evaluation Engine
• Components: src/compute.py (orchestrator), src/computeScore.py (metrics), packages/ (test harness)
• Reads JSON from data/, uses packages to instantiate input structures, runs each solution in a sandbox (time.time(), tracemalloc), produces per-test metrics
c. Results Storage & Visualization
• Components: data → csv files in results/ (per model), results/visualisation.py generates charts or summary tables
d. Containerization & CLI
• Components: Dockerfile (+ .dockerignore) encapsulates the entire pipeline in a reproducible environment
• Usage: python src/openSourceLLMoutput.py → python src/compute.py
Architectural patterns & design principles
• Pipeline / ETL-style flow: Inference → Storage → Processing → Reporting
• Modular separation: distinct modules for API calls, testing logic, data models (packages), and visualization
• Single-responsibility: each script or module has a clear role
• Configurable prompts & imports (prompt.py, imports.txt) to support extensibility