How dineshdeepanshu2002/rag-eval-thesis works
This repository is a reproducible research framework comparing RAG evaluation methods across datasets and system configurations. The main workflow expands experiment settings, loads and samples BEIR data, chunks documents, retrieves passages, generates answers, scores outputs with IR, RAGAS, and ARES, then aggregates results and analyzes agreement and configuration effects. A separate human-labeling workflow supplies judgments for ARES calibration. The README documents these capabilities; source excerpts show key implementation stages, but some orchestration and output-writing details were not sampled.
Main entry points: run_experiment.py Experiment runner · label_cli.py Human label tool · human_labels Human labels · loaders.py BEIR data
Components 18
Experiment workflow
- Experiment runner
scripts/run_experiment.py - Config matrix
src/config.py - BEIR data
src/data/loaders.py - Label-sheet creation
scripts/make_annotation_sheet.py - Human label tool
scripts/label_cli.py
RAG pipeline
- Document chunking
src/pipeline/chunking.py - Retrieval and reranking
src/pipeline/retrieval.py - Answer generation
src/pipeline/generation.py
Evaluation
- IR metrics
src/evaluation/ir_metrics.py - RAGAS evaluation
src/evaluation/ragas_eval.py - ARES judging and PPI
src/evaluation/ares_eval.py
Calibration and analysis
- Human labels
results/human_labels - Agreement statistics
src/stats/agreement.py - Experiment results
results
Other components
- Researcher
- Human annotator
- BEIR datasets
- OpenAI models
Connections 22
- Researcher → Experiment runner: runs experiments
README.md - Researcher → Label-sheet creation: creates sheets
README.md - Researcher → Human label tool: labels examples
README.md - Experiment runner → Config matrix: loads settings
scripts/run_experiment.py - Experiment runner → BEIR data: loads dataset
scripts/run_experiment.py - BEIR data → BEIR datasets: downloads data
src/data/loaders.py - Experiment runner → Document chunking: chunks corpus
scripts/run_experiment.py - Experiment runner → Retrieval and reranking: retrieves passages
scripts/run_experiment.py - Experiment runner → Answer generation: generates answers
scripts/run_experiment.py - Answer generation → OpenAI models: requests completions
src/pipeline/generation.py - Experiment runner → IR metrics: scores retrieval
scripts/run_experiment.py - Experiment runner → RAGAS evaluation: scores answers
scripts/run_experiment.py - RAGAS evaluation → OpenAI models: judges responses
src/evaluation/ragas_eval.py - Experiment runner → ARES judging and PPI: judges and calibrates
scripts/run_experiment.py - ARES judging and PPI → OpenAI models: requests judgments
src/evaluation/ares_eval.py - Label-sheet creation → Human labels: prepares label sheets
README.md - Human label tool → Human labels: writes judgments
scripts/label_cli.py - Human annotator → Human label tool: provides judgments
scripts/label_cli.py - Human labels → Experiment runner: supplies calibration data
scripts/run_experiment.py - Experiment runner → Agreement statistics: analyzes scores
scripts/run_experiment.py - Experiment runner → Experiment results: writes results
README.md - Agreement statistics → Experiment results: produces analysis tables
README.md