How sam22ridhi/eka-eval works
Project Type and Purpose
• Type: Open-source evaluation framework (Python library + CLI)
• Purpose: Provide a unified, extensible pipeline for benchmarking large language models (LLMs) on global and India-centric tasks, with multi-GPU support, quantization, prompt templates, and detailed reporting.
File Structure
• Top-level folders:
– eka_eval/ (core library)
◦ benchmarks/ (all evaluation tasks, grouped by category)
◦ core/ (model loading, evaluator orchestration)
◦ config/ (JSON-based benchmark registry/configuration)
◦ utils/ (logging, file, GPU helpers)
– scripts/ (CLI entrypoints & worker processes)
– prompts/ (prompt templates organized by task category)
– results_output/ (evaluation outputs)
– tests/ (unit tests)
• Key standalone files:
– run_benchmarks.py (top-level launcher)
– pyproject.toml / requirements.txt (build & dependencies)
• Patterns:
– Modular “plugin” style: benchmarks & prompts discoverable by registry
– CLI → script workers → core evaluator → model loader → dataset/tasks → prompt templates → huggingface pipelines → result sinks
README for Additional Insights
• Architecture & Stack: Python, PyTorch, Transformers, Accelerate, Hugging Face, BitsAndBytes (quantization)
• Features: Multi-GPU dispatch, batched inference, JSON config registry, prompt templating, CSV/JSONL/console outputs
• Usage Flow: Interactive CLI → select tasks/models → spawn worker processes → load model → evaluate benchmarks → write results
a. Main Components
1. CLI Layer
• scripts/run_benchmarks.py / eka_eval/scripts/main.py
• Interactive prompts, argument parsing
2. Orchestrator / Dispatcher
• Spawns evaluation_worker.py processes (via multi-processing or Accelerate)
• Distributes GPU devices
3. Core Library
• Model Loader (core/model_loader.py): loads HF models with quantization flags
• Evaluator (core/evaluator.py): loops over tasks, invokes task-specific eval fns
• Benchmark Registry (benchmarks/benchmark_registry.py + config/benchmark_config.py): maps config → evaluation functions & prompt keys
4. Benchmark Modules (benchmarks/tasks/...)
• Code, Math, Commonsense, Indic, Reading, Long Context
• Each implements an evaluate(...) function
5. Prompt Store
• JSON files under prompts/
• Templates loaded by registry, filled per example
6. Inference Engine
• Hugging Face transformers.Pipeline
• Batched calls to the model on GPU
7. Utilities
• Logging, file I/O, GPU management, result formatting
8. Result Sink
• Writes CSV (results_output/calculated.csv), JSONL details, console tables
b. Relationships & Data Flows
1. User → CLI Layer
2. CLI → Orchestrator → spawn Worker(s)
3. Worker → Core Library
• Core loads Benchmark Registry & Config
• Core requests Model Loader to instantiate the model pipeline
4. Core → Benchmark Modules (for each task):
• Loads prompt template → builds inputs → invokes Inference Engine
• Collects raw outputs → passes to evaluator → computes metrics
5. Core → Utilities / Logger → progress logs
6. Core → Result Sink → writes to CSV/JSONL
c. Architectural Patterns & Principles
• Modularity & Extensibility: Registry-based plugin pattern for adding new benchmarks or languages
• Separation of Concerns: Clear layers (CLI, orchestration, core logic, tasks, I/O)
• Scalability: Multi-process / multi-GPU dispatch, batched inference
• Configurability: JSON / CLI driven, environment-based quantization & device selection
d. Key Technologies & Frameworks
• Python 3.x, PyTorch, Transformers, Accelerate, BitsAndBytes
• Hugging Face Datasets/Evaluate for metrics
• Pandas, NumPy, tqdm for reporting & progress
• JSON for prompt templates & config
• Multiprocessing / Torch Distributed / Accelerate for parallelism
Detail Without Over-generalizing
• Break out each of the 6 major subsystems (CLI, Orchestrator, Core, Tasks, Inference, Results)
• Registry pattern: Config → Registry → dynamic loading of evaluation functions & prompt keys
• Parallelism: Worker processes and GPU assignments
• Don’t forget utilities (logging, GPU management) as side-components feeding into core