# megh-zyke/efficiency-benchmark architecture

> Architecture diagram of the GitHub repository megh-zyke/efficiency-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/megh-zyke/efficiency-benchmark
- Repository: https://github.com/megh-zyke/efficiency-benchmark

## Overview

##### Project type and purpose
- This is a Python-based benchmarking framework that generates, executes, and scores LLM-written algorithmic code for efficiency (time + memory) and correctness.
- It drives multiple LLM providers (Hugging Face models, Groq, Gemini, etc.), collects their outputs, runs stress tests, computes unified scores, and produces CSV results + visualizations.

##### File structure
- Top-level scripts & config:
- Dockerfile / .dockerignore – containerizes the environment
- requirements.txt – Python dependencies
- README.md – project overview
- data/ – stores raw JSON outputs from various LLMs (e.g. gemini-2.5-pro.json)
- src/ – core pipeline code:
- openSourceLLMoutput.py & groqLLMoutput.py – fetch code from external LLM APIs
- compute.py & computeScore.py – run tests, measure time/memory, calculate scores
- imports.txt – shared import definitions
- packages/ – utility classes for constructing test inputs & parsing outputs (ListNode, TreeNode, etc.)
- results/ – stores per-model CSV results & a visualization script (visualisation.py)

##### README
- Confirms pipeline stages: LLM inference → data storage → compute metrics → results + plots
- Lists evaluation metrics (correctness, execution time, memory) and scoring formula
- Mentions stress-testing levels and dataset sources

##### Main components & their relationships
a. LLM Inference Layer
- Components: openSourceLLMoutput.py, groqLLMoutput.py, prompts/prompt.py
- External Services: Hugging Face API, Groq LLM endpoint, other LLM providers
- Output: JSON files in data/ directory
b. Evaluation Engine
- Components: src/compute.py (orchestrator), src/computeScore.py (metrics), packages/ (test harness)
- Reads JSON from data/, uses packages to instantiate input structures, runs each solution in a sandbox (time.time(), tracemalloc), produces per-test metrics
c. Results Storage & Visualization
- Components: data → csv files in results/ (per model), results/visualisation.py generates charts or summary tables
d. Containerization & CLI
- Components: Dockerfile (+ .dockerignore) encapsulates the entire pipeline in a reproducible environment
- Usage: `python src/openSourceLLMoutput.py` → `python src/compute.py`

##### Architectural patterns & design principles
- Pipeline / ETL-style flow: Inference → Storage → Processing → Reporting
- Modular separation: distinct modules for API calls, testing logic, data models (packages), and visualization
- Single-responsibility: each script or module has a clear role
- Configurable prompts & imports (prompt.py, imports.txt) to support extensibility

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Inference Layer"
        prompt["Prompt Manager"]:::internal
        oss["Open-Source LLM Client"]:::internal
        groq["Groq LLM Client"]:::internal
    end

    subgraph "External LLM Services"
        hf["Hugging Face API"]:::external
        groqSrv["Groq Endpoint"]:::external
    end

    dataStore["Data Store (JSON)"]:::data

    subgraph "Evaluation Engine"
        orchestrator["Evaluation Engine Orchestrator"]:::internal
        metrics["Evaluation Metrics Calculator"]:::internal
        testHarness["Test Harness & Data Models"]:::internal
        imports["Shared Imports Config"]:::config
    end

    subgraph "Results & Visualization"
        resultsStore["Results Store (CSV)"]:::data
        viz["Visualization Module"]:::internal
    end

    container["Container / CLI"]:::cli
    deps["Dependency Spec (requirements.txt)"]:::config
    readme["Project Overview (README.md)"]:::config

    prompt -->|"feed prompts"| oss
    prompt -->|"feed prompts"| groq
    oss -->|"HTTP/API calls"| hf
    groq -->|"HTTP/API calls"| groqSrv
    hf -->|"JSON responses"| oss
    groqSrv -->|"JSON responses"| groq
    oss -->|".json"| dataStore
    groq -->|".json"| dataStore
    dataStore -->|"read JSON"| orchestrator
    orchestrator --> metrics
    orchestrator --> testHarness
    orchestrator --> imports
    metrics -->|".csv"| resultsStore
    resultsStore -->|".csv"| viz
    testHarness --> orchestrator

    container --> oss
    container --> orchestrator
    container --> viz
    container --> prompt
    deps --> container
    readme --> container

    click prompt "https://github.com/megh-zyke/efficiency-benchmark/blob/main/prompts/prompt.py"
    click oss "https://github.com/megh-zyke/efficiency-benchmark/blob/main/src/openSourceLLMoutput.py"
    click groq "https://github.com/megh-zyke/efficiency-benchmark/blob/main/src/groqLLMoutput.py"
    click dataStore "https://github.com/megh-zyke/efficiency-benchmark/tree/main/data/"
    click orchestrator "https://github.com/megh-zyke/efficiency-benchmark/blob/main/src/compute.py"
    click metrics "https://github.com/megh-zyke/efficiency-benchmark/blob/main/src/computeScore.py"
    click testHarness "https://github.com/megh-zyke/efficiency-benchmark/tree/main/packages/"
    click imports "https://github.com/megh-zyke/efficiency-benchmark/blob/main/src/imports.txt"
    click resultsStore "https://github.com/megh-zyke/efficiency-benchmark/tree/main/results/"
    click viz "https://github.com/megh-zyke/efficiency-benchmark/blob/main/results/visualisation.py"
    click container "https://github.com/megh-zyke/efficiency-benchmark/tree/main/Dockerfile"
    click deps "https://github.com/megh-zyke/efficiency-benchmark/blob/main/requirements.txt"
    click readme "https://github.com/megh-zyke/efficiency-benchmark/blob/main/README.md"

    classDef internal fill:#ADD8E6,stroke:#333,stroke-width:2px
    classDef data fill:#90EE90,stroke:#333,stroke-width:2px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef cli fill:#F0E68C,stroke:#333,stroke-width:2px
    classDef config fill:#DDA0DD,stroke:#333,stroke-width:2px
```
