# vinodbavage31/rag-evaluation-framework architecture

> Architecture diagram of the GitHub repository vinodbavage31/rag-evaluation-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vinodbavage31/rag-evaluation-framework
- Repository: https://github.com/vinodbavage31/rag-evaluation-framework

## Overview

WMF RAG Evaluation Framework evaluates stored RAG traces against a validated Golden Dataset; it does not execute RAG. The principal workflow is the runner loading cases, retrieving traces (or offline fixtures), reconstructing actual results, evaluating metrics, and writing reports. The graph preserves deterministic metrics, RAGAS, and structured Gemini judging as distinct capabilities. Edges reflect the README and shown caller wiring; unsampled implementation details are not inferred.

Main entry points: [`scripts/run_evaluation.py`](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/scripts/run_evaluation.py) (Evaluation runner), [`src/golden_dataset.py`](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/golden_dataset.py) (Golden Dataset), [`src/trace_database.py`](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_database.py) (Read-only DB engine), [`src/trace_repository.py`](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_repository.py) (Trace repository)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Run orchestration"]
  node_runner["Evaluation runner<br/>[run_evaluation.py]"]
  node_engine["Read-only DB engine<br/>[trace_database.py]"]
  node_repository["Trace repository"]
  node_reconstructor["Trace reconstructor"]
  node_actual["Actual RAG result"]
end

subgraph group_inputs["Evaluation inputs"]
  node_config["Evaluation config<br/>[evaluation.yaml]"]
  node_golden["Golden Dataset<br/>[golden_dataset.py]"]
  node_dataset_file["Dataset records"]
end

subgraph group_analysis["Evaluation methods"]
  node_deterministic["Deterministic metrics<br/>[evaluator.py]"]
  node_ragas["RAGAS metrics<br/>[ragas_evaluator.py]"]
  node_judge["Gemini judge<br/>[gemini_judge.py]"]
end

subgraph group_outputs["Reports and storage"]
  node_reporter["Report generator<br/>[reporting.py]"]
  node_reports["Run reports"]
end

node_operator(("Operator"))
node_postgres[("Trace database")]
node_gemini{{"Gemini API"}}

node_operator -->|"runs"| node_runner
node_runner -->|"uses"| node_config
node_runner -->|"loads cases"| node_golden
node_golden -->|"reads settings"| node_config
node_golden -->|"loads records"| node_dataset_file
node_runner -->|"creates engine"| node_engine
node_engine -->|"connects read-only"| node_postgres
node_runner -->|"fetches traces"| node_repository
node_repository -->|"selects trace rows"| node_postgres
node_runner -->|"reconstructs traces"| node_reconstructor
node_reconstructor -->|"produces"| node_actual
node_runner -->|"computes metrics"| node_deterministic
node_deterministic -->|"compares results"| node_actual
node_runner -->|"requests scores"| node_ragas
node_ragas -->|"calls model"| node_gemini
node_runner -->|"requests judgment"| node_judge
node_judge -->|"calls model"| node_gemini
node_runner -->|"writes reports"| node_reporter
node_reporter -->|"writes JSON CSV HTML"| node_reports

click node_runner "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/scripts/run_evaluation.py"
click node_config "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/config/evaluation.yaml"
click node_golden "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/golden_dataset.py"
click node_dataset_file "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/datasets/golden/golden_dataset.json"
click node_engine "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_database.py"
click node_repository "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_repository.py"
click node_reconstructor "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_reconstructor.py"
click node_actual "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_reconstructor.py"
click node_deterministic "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/evaluator.py"
click node_ragas "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/ragas_evaluator.py"
click node_judge "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/gemini_judge.py"
click node_reporter "https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/reporting.py"
click node_reports "https://github.com/vinodbavage31/rag-evaluation-framework/tree/main/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_engine,node_repository,node_reconstructor,node_actual toneBlue
class node_config,node_golden,node_dataset_file,node_postgres toneAmber
class node_deterministic,node_ragas,node_judge,node_gemini toneMint
class node_reporter,node_reports toneRose
class node_operator toneIndigo
```

## Components

### Run orchestration

- [Evaluation runner](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/scripts/run_evaluation.py): `scripts/run_evaluation.py`
- [Read-only DB engine](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_database.py): `src/trace_database.py`
- [Trace repository](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_repository.py): `src/trace_repository.py`
- [Trace reconstructor](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_reconstructor.py): `src/trace_reconstructor.py`
- [Actual RAG result](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/trace_reconstructor.py): `src/trace_reconstructor.py`

### Evaluation inputs

- [Evaluation config](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/config/evaluation.yaml): `config/evaluation.yaml`
- [Golden Dataset](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/golden_dataset.py): `src/golden_dataset.py`
- [Dataset records](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/datasets/golden/golden_dataset.json): `datasets/golden/golden_dataset.json`

### Evaluation methods

- [Deterministic metrics](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/evaluator.py): `src/evaluator.py`
- [RAGAS metrics](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/ragas_evaluator.py): `src/ragas_evaluator.py`
- [Gemini judge](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/gemini_judge.py): `src/gemini_judge.py`

### Reports and storage

- [Report generator](https://github.com/vinodbavage31/rag-evaluation-framework/blob/main/src/reporting.py): `src/reporting.py`
- [Run reports](https://github.com/vinodbavage31/rag-evaluation-framework/tree/main/results): `results`

### Other components

- **Operator**
- **Trace database**
- **Gemini API**

## Connections

- Operator → Evaluation runner: runs (evidence: `README.md`)
- Evaluation runner → Evaluation config: uses (evidence: `README.md`)
- Evaluation runner → Golden Dataset: loads cases (evidence: `scripts/run_evaluation.py`)
- Golden Dataset → Evaluation config: reads settings (evidence: `README.md`)
- Golden Dataset → Dataset records: loads records (evidence: `README.md`)
- Evaluation runner → Read-only DB engine: creates engine (evidence: `scripts/run_evaluation.py`)
- Read-only DB engine → Trace database: connects read-only (evidence: `README.md`)
- Evaluation runner → Trace repository: fetches traces (evidence: `scripts/run_evaluation.py`)
- Trace repository → Trace database: selects trace rows (evidence: `src/trace_repository.py`)
- Evaluation runner → Trace reconstructor: reconstructs traces (evidence: `scripts/run_evaluation.py`)
- Trace reconstructor → Actual RAG result: produces (evidence: `src/trace_reconstructor.py`)
- Evaluation runner → Deterministic metrics: computes metrics (evidence: `scripts/run_evaluation.py`)
- Deterministic metrics → Actual RAG result: compares results (evidence: `src/evaluator.py`)
- Evaluation runner → RAGAS metrics: requests scores (evidence: `scripts/run_evaluation.py`)
- RAGAS metrics → Gemini API: calls model (evidence: `README.md`)
- Evaluation runner → Gemini judge: requests judgment (evidence: `scripts/run_evaluation.py`)
- Gemini judge → Gemini API: calls model (evidence: `src/gemini_judge.py`)
- Evaluation runner → Report generator: writes reports (evidence: `scripts/run_evaluation.py`)
- Report generator → Run reports: writes JSON CSV HTML (evidence: `README.md`)
