# sam22ridhi/eka-eval architecture

> Architecture diagram of the GitHub repository sam22ridhi/eka-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sam22ridhi/eka-eval
- Repository: https://github.com/sam22ridhi/eka-eval

## Overview

##### Project Type and Purpose
- **Type**: Open-source evaluation framework (Python library + CLI)
- **Purpose**: Provide a unified, extensible pipeline for benchmarking large language models (LLMs) on global and India-centric tasks, with multi-GPU support, quantization, prompt templates, and detailed reporting.

##### File Structure
- Top-level folders:
– `eka_eval/` (core library)
◦ `benchmarks/` (all evaluation tasks, grouped by category)
◦ `core/` (model loading, evaluator orchestration)
◦ `config/` (JSON-based benchmark registry/configuration)
◦ `utils/` (logging, file, GPU helpers)
– `scripts/` (CLI entrypoints & worker processes)
– `prompts/` (prompt templates organized by task category)
– `results_output/` (evaluation outputs)
– `tests/` (unit tests)
- Key standalone files:
– `run_benchmarks.py` (top-level launcher)
– `pyproject.toml` / `requirements.txt` (build & dependencies)
- Patterns:
– Modular “plugin” style: benchmarks & prompts discoverable by registry
– CLI → script workers → core evaluator → model loader → dataset/tasks → prompt templates → huggingface pipelines → result sinks

##### README for Additional Insights
- **Architecture & Stack**: Python, PyTorch, Transformers, Accelerate, Hugging Face, BitsAndBytes (quantization)
- **Features**: Multi-GPU dispatch, batched inference, JSON config registry, prompt templating, CSV/JSONL/console outputs
- **Usage Flow**: Interactive CLI → select tasks/models → spawn worker processes → load model → evaluate benchmarks → write results

a. **Main Components**
1. **CLI Layer**
- `scripts/run_benchmarks.py` / `eka_eval/scripts/main.py`
- Interactive prompts, argument parsing
2. **Orchestrator / Dispatcher**
- Spawns `evaluation_worker.py` processes (via multi-processing or Accelerate)
- Distributes GPU devices
3. **Core Library**
- **Model Loader** (`core/model_loader.py`): loads HF models with quantization flags
- **Evaluator** (`core/evaluator.py`): loops over tasks, invokes task-specific eval fns
- **Benchmark Registry** (`benchmarks/benchmark_registry.py` + `config/benchmark_config.py`): maps config → evaluation functions & prompt keys
4. **Benchmark Modules** (`benchmarks/tasks/...`)
- Code, Math, Commonsense, Indic, Reading, Long Context
- Each implements an `evaluate(...)` function
5. **Prompt Store**
- JSON files under `prompts/`
- Templates loaded by registry, filled per example
6. **Inference Engine**
- Hugging Face `transformers.Pipeline`
- Batched calls to the model on GPU
7. **Utilities**
- Logging, file I/O, GPU management, result formatting
8. **Result Sink**
- Writes CSV (`results_output/calculated.csv`), JSONL details, console tables
b. **Relationships & Data Flows**
1. **User** → CLI Layer
2. CLI → Orchestrator → spawn Worker(s)
3. Worker → Core Library
- Core loads Benchmark Registry & Config
- Core requests Model Loader to instantiate the model pipeline
4. Core → Benchmark Modules (for each task):
- Loads prompt template → builds inputs → invokes Inference Engine
- Collects raw outputs → passes to evaluator → computes metrics
5. Core → Utilities / Logger → progress logs
6. Core → Result Sink → writes to CSV/JSONL
c. **Architectural Patterns & Principles**
- **Modularity & Extensibility**: Registry-based plugin pattern for adding new benchmarks or languages
- **Separation of Concerns**: Clear layers (CLI, orchestration, core logic, tasks, I/O)
- **Scalability**: Multi-process / multi-GPU dispatch, batched inference
- **Configurability**: JSON / CLI driven, environment-based quantization & device selection
d. **Key Technologies & Frameworks**
- Python 3.x, PyTorch, Transformers, Accelerate, BitsAndBytes
- Hugging Face Datasets/Evaluate for metrics
- Pandas, NumPy, tqdm for reporting & progress
- JSON for prompt templates & config
- Multiprocessing / Torch Distributed / Accelerate for parallelism

##### Detail Without Over-generalizing
- Break out each of the 6 major subsystems (CLI, Orchestrator, Core, Tasks, Inference, Results)
- Registry pattern: Config → Registry → dynamic loading of evaluation functions & prompt keys
- Parallelism: Worker processes and GPU assignments
- Don’t forget utilities (logging, GPU management) as side-components feeding into core

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Interaction
    User["User"]:::external
    User -->|"starts CLI"| CLI_Main

    %% CLI Layer
    subgraph "CLI Layer"
        CLI_Main["eka_eval/scripts/main.py"]:::service
        CLI_RunScript["eka_eval/scripts/run_benchmarks.py"]:::service
        CLI_TopRun["run_benchmarks.py"]:::service
    end
    User --> CLI_TopRun
    CLI_Main --> CLI_RunScript
    CLI_RunScript --> Orchestrator

    %% Orchestrator / Dispatcher
    Orchestrator["Orchestrator / Dispatcher"]:::service
    Orchestrator -->|"spawn processes"| WorkerProcess
    WorkerProcess["evaluation_worker.py"]:::service
    WorkerProcess --> CoreLibrary

    %% Core Library
    subgraph "Core Library"
        ModelLoader["Model Loader"]:::service
        Evaluator["Evaluator"]:::service
        Registry["Benchmark Registry"]:::service
        ConfigStore["Benchmark Config"]:::datastore
    end
    CoreLibrary[/"Core Library"/]:::component
    WorkerProcess --> CoreLibrary
    CoreLibrary --> Registry
    CoreLibrary --> ConfigStore
    CoreLibrary --> ModelLoader
    CoreLibrary --> Evaluator

    %% Benchmark Modules
    subgraph "Benchmark Modules"
        CodeTask["Code"]:::component
        Commonsense["Commonsense Reasoning"]:::component
        MathTask["Math Eval"]:::component
        ReadingTask["Reading Comprehension"]:::component
    end
    Evaluator --> CodeTask
    Evaluator --> Commonsense
    Evaluator --> MathTask
    Evaluator --> ReadingTask

    %% Prompt Store
    subgraph "Prompt Store"
        Prompt_Code["prompts/code/"]:::datastore
        Prompt_Common["prompts/commonsense/"]:::datastore
        Prompt_Math["prompts/math/"]:::datastore
        Prompt_Reading["prompts/reading/"]:::datastore
        Prompt_Indic["prompts/indic/"]:::datastore
        Prompt_Long["prompts/long_context/"]:::datastore
        Prompt_General["prompts/general/"]:::datastore
    end
    Registry --> Prompt_Code
    Registry --> Prompt_Common
    Registry --> Prompt_Math
    Registry --> Prompt_Reading
    Registry --> Prompt_Indic
    Registry --> Prompt_Long
    Registry --> Prompt_General

    %% Inference Engine
    Inference["Inference Engine\n(HF Transformers Pipeline)"]:::external
    ModelLoader --> Inference
    CodeTask --> Inference
    Commonsense --> Inference
    MathTask --> Inference
    ReadingTask --> Inference

    %% Utilities
    subgraph "Utilities"
        LoggingSetup["logging_setup.py"]:::util
        FileUtils["file_utils.py"]:::util
        GPUUtils["gpu_utils.py"]:::util
        Constants["constants.py"]:::util
    end
    CoreLibrary --> LoggingSetup
    CoreLibrary --> FileUtils
    CoreLibrary --> GPUUtils
    CoreLibrary --> Constants

    %% Result Sink
    Results["Result Sink"]:::datastore
    Evaluator --> Results
    Results -->|"writes CSV/JSONL"| ResultsCSV

    ResultsCSV["results_output/calculated.csv"]:::datastore

    %% External Services
    HFHub["Hugging Face Hub"]:::external
    GPUs["GPUs (CUDA)"]:::external
    Inference --> HFHub
    WorkerProcess --> GPUs

    %% Dependency & Build
    subgraph "Build Config"
        PyProject["pyproject.toml"]:::datastore
        Requirements["requirements.txt"]:::datastore
    end
    User --> PyProject
    User --> Requirements

    %% Styles
    classDef service fill:#D6EFFF,stroke:#000;
    classDef component fill:#AED6F1,stroke:#000;
    classDef datastore fill:#D5F5E3,stroke:#000;
    classDef external fill:#FADBD8,stroke:#000;
    classDef util fill:#EBF5FB,stroke:#000;

    %% Click Events
    click CLI_Main "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/scripts/main.py"
    click CLI_RunScript "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/scripts/run_benchmarks.py"
    click CLI_TopRun "https://github.com/sam22ridhi/eka-eval/blob/main/run_benchmarks.py"
    click WorkerProcess "https://github.com/sam22ridhi/eka-eval/blob/main/scripts/evaluation_worker.py"
    click ModelLoader "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/core/model_loader.py"
    click Evaluator "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/core/evaluator.py"
    click Registry "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/benchmarks/benchmark_registry.py"
    click ConfigStore "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/config/benchmark_config.py"
    click CodeTask "https://github.com/sam22ridhi/eka-eval/tree/main/eka_eval/benchmarks/tasks/code/"
    click Commonsense "https://github.com/sam22ridhi/eka-eval/tree/main/eka_eval/benchmarks/tasks/commonsense_reasoning/"
    click MathTask "https://github.com/sam22ridhi/eka-eval/tree/main/eka_eval/benchmarks/tasks/math_eval/"
    click ReadingTask "https://github.com/sam22ridhi/eka-eval/tree/main/eka_eval/benchmarks/tasks/reading_comprehension/"
    click Prompt_Code "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/code/"
    click Prompt_Common "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/commonsense/"
    click Prompt_Math "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/math/"
    click Prompt_Reading "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/reading/"
    click Prompt_Indic "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/indic/"
    click Prompt_Long "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/long_context/"
    click Prompt_General "https://github.com/sam22ridhi/eka-eval/tree/main/prompts/general/"
    click LoggingSetup "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/utils/logging_setup.py"
    click FileUtils "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/utils/file_utils.py"
    click GPUUtils "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/utils/gpu_utils.py"
    click Constants "https://github.com/sam22ridhi/eka-eval/blob/main/eka_eval/utils/constants.py"
    click ResultsCSV "https://github.com/sam22ridhi/eka-eval/blob/main/results_output/calculated.csv"
    click PyProject "https://github.com/sam22ridhi/eka-eval/blob/main/pyproject.toml"
    click Requirements "https://github.com/sam22ridhi/eka-eval/blob/main/requirements.txt"
```
