# dineshdeepanshu2002/rag-eval-thesis architecture

> Architecture diagram of the GitHub repository dineshdeepanshu2002/rag-eval-thesis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dineshdeepanshu2002/rag-eval-thesis
- Repository: https://github.com/dineshdeepanshu2002/rag-eval-thesis

## Overview

This repository is a reproducible research framework comparing RAG evaluation methods across datasets and system configurations. The main workflow expands experiment settings, loads and samples BEIR data, chunks documents, retrieves passages, generates answers, scores outputs with IR, RAGAS, and ARES, then aggregates results and analyzes agreement and configuration effects. A separate human-labeling workflow supplies judgments for ARES calibration. The README documents these capabilities; source excerpts show key implementation stages, but some orchestration and output-writing details were not sampled.

Main entry points: [`scripts/run_experiment.py`](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/run_experiment.py) (Experiment runner), [`scripts/label_cli.py`](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/label_cli.py) (Human label tool), [`results/human_labels`](https://github.com/dineshdeepanshu2002/rag-eval-thesis/tree/main/results/human_labels) (Human labels), [`src/data/loaders.py`](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/data/loaders.py) (BEIR data)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experiment["Experiment workflow"]
  node_orchestrator["Experiment runner<br/>[run_experiment.py]"]
  node_config["Config matrix<br/>[config.py]"]
  node_dataset["BEIR data<br/>[loaders.py]"]
  node_annotation["Label-sheet creation"]
  node_labeler["Human label tool<br/>[label_cli.py]"]
end

subgraph group_rag["RAG pipeline"]
  node_chunking["Document chunking<br/>[chunking.py]"]
  node_retrieval["Retrieval and reranking<br/>[retrieval.py]"]
  node_generation["Answer generation<br/>[generation.py]"]
end

subgraph group_scoring["Evaluation"]
  node_ir["IR metrics<br/>[ir_metrics.py]"]
  node_ragas["RAGAS evaluation<br/>[ragas_eval.py]"]
  node_ares["ARES judging and PPI<br/>[ares_eval.py]"]
end

subgraph group_analysis["Calibration and analysis"]
  node_labels[("Human labels")]
  node_agreement["Agreement statistics<br/>[agreement.py]"]
  node_outputs[("Experiment results")]
end

node_operator(("Researcher"))
node_human(("Human annotator"))
node_beir[("BEIR datasets")]
node_llm["OpenAI models"]

node_operator -->|"runs experiments"| node_orchestrator
node_operator -->|"creates sheets"| node_annotation
node_operator -->|"labels examples"| node_labeler
node_orchestrator -->|"loads settings"| node_config
node_orchestrator -->|"loads dataset"| node_dataset
node_dataset -->|"downloads data"| node_beir
node_orchestrator -->|"chunks corpus"| node_chunking
node_orchestrator -->|"retrieves passages"| node_retrieval
node_orchestrator -->|"generates answers"| node_generation
node_generation -->|"requests completions"| node_llm
node_orchestrator -->|"scores retrieval"| node_ir
node_orchestrator -->|"scores answers"| node_ragas
node_ragas -->|"judges responses"| node_llm
node_orchestrator -->|"judges and calibrates"| node_ares
node_ares -->|"requests judgments"| node_llm
node_annotation -->|"prepares label sheets"| node_labels
node_labeler -->|"writes judgments"| node_labels
node_human -->|"provides judgments"| node_labeler
node_labels -->|"supplies calibration data"| node_orchestrator
node_orchestrator -->|"analyzes scores"| node_agreement
node_orchestrator -->|"writes results"| node_outputs
node_agreement -->|"produces analysis tables"| node_outputs

click node_orchestrator "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/run_experiment.py"
click node_config "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/config.py"
click node_dataset "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/data/loaders.py"
click node_chunking "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/pipeline/chunking.py"
click node_retrieval "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/pipeline/retrieval.py"
click node_generation "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/pipeline/generation.py"
click node_ir "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/evaluation/ir_metrics.py"
click node_ragas "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/evaluation/ragas_eval.py"
click node_ares "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/evaluation/ares_eval.py"
click node_annotation "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/make_annotation_sheet.py"
click node_labeler "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/label_cli.py"
click node_labels "https://github.com/dineshdeepanshu2002/rag-eval-thesis/tree/main/results/human_labels"
click node_agreement "https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/stats/agreement.py"
click node_outputs "https://github.com/dineshdeepanshu2002/rag-eval-thesis/tree/main/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_config,node_dataset,node_annotation,node_labeler toneBlue
class node_chunking,node_retrieval,node_generation,node_beir toneAmber
class node_ir,node_ragas,node_ares toneMint
class node_labels,node_agreement,node_outputs toneRose
class node_operator,node_human,node_llm toneIndigo
```

## Components

### Experiment workflow

- [Experiment runner](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/run_experiment.py): `scripts/run_experiment.py`
- [Config matrix](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/config.py): `src/config.py`
- [BEIR data](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/data/loaders.py): `src/data/loaders.py`
- [Label-sheet creation](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/make_annotation_sheet.py): `scripts/make_annotation_sheet.py`
- [Human label tool](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/scripts/label_cli.py): `scripts/label_cli.py`

### RAG pipeline

- [Document chunking](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/pipeline/chunking.py): `src/pipeline/chunking.py`
- [Retrieval and reranking](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/pipeline/retrieval.py): `src/pipeline/retrieval.py`
- [Answer generation](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/pipeline/generation.py): `src/pipeline/generation.py`

### Evaluation

- [IR metrics](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/evaluation/ir_metrics.py): `src/evaluation/ir_metrics.py`
- [RAGAS evaluation](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/evaluation/ragas_eval.py): `src/evaluation/ragas_eval.py`
- [ARES judging and PPI](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/evaluation/ares_eval.py): `src/evaluation/ares_eval.py`

### Calibration and analysis

- [Human labels](https://github.com/dineshdeepanshu2002/rag-eval-thesis/tree/main/results/human_labels): `results/human_labels`
- [Agreement statistics](https://github.com/dineshdeepanshu2002/rag-eval-thesis/blob/main/src/stats/agreement.py): `src/stats/agreement.py`
- [Experiment results](https://github.com/dineshdeepanshu2002/rag-eval-thesis/tree/main/results): `results`

### Other components

- **Researcher**
- **Human annotator**
- **BEIR datasets**
- **OpenAI models**

## Connections

- Researcher → Experiment runner: runs experiments (evidence: `README.md`)
- Researcher → Label-sheet creation: creates sheets (evidence: `README.md`)
- Researcher → Human label tool: labels examples (evidence: `README.md`)
- Experiment runner → Config matrix: loads settings (evidence: `scripts/run_experiment.py`)
- Experiment runner → BEIR data: loads dataset (evidence: `scripts/run_experiment.py`)
- BEIR data → BEIR datasets: downloads data (evidence: `src/data/loaders.py`)
- Experiment runner → Document chunking: chunks corpus (evidence: `scripts/run_experiment.py`)
- Experiment runner → Retrieval and reranking: retrieves passages (evidence: `scripts/run_experiment.py`)
- Experiment runner → Answer generation: generates answers (evidence: `scripts/run_experiment.py`)
- Answer generation → OpenAI models: requests completions (evidence: `src/pipeline/generation.py`)
- Experiment runner → IR metrics: scores retrieval (evidence: `scripts/run_experiment.py`)
- Experiment runner → RAGAS evaluation: scores answers (evidence: `scripts/run_experiment.py`)
- RAGAS evaluation → OpenAI models: judges responses (evidence: `src/evaluation/ragas_eval.py`)
- Experiment runner → ARES judging and PPI: judges and calibrates (evidence: `scripts/run_experiment.py`)
- ARES judging and PPI → OpenAI models: requests judgments (evidence: `src/evaluation/ares_eval.py`)
- Label-sheet creation → Human labels: prepares label sheets (evidence: `README.md`)
- Human label tool → Human labels: writes judgments (evidence: `scripts/label_cli.py`)
- Human annotator → Human label tool: provides judgments (evidence: `scripts/label_cli.py`)
- Human labels → Experiment runner: supplies calibration data (evidence: `scripts/run_experiment.py`)
- Experiment runner → Agreement statistics: analyzes scores (evidence: `scripts/run_experiment.py`)
- Experiment runner → Experiment results: writes results (evidence: `README.md`)
- Agreement statistics → Experiment results: produces analysis tables (evidence: `README.md`)
