How vinodbavage31/rag-evaluation-framework works
WMF RAG Evaluation Framework evaluates stored RAG traces against a validated Golden Dataset; it does not execute RAG. The principal workflow is the runner loading cases, retrieving traces (or offline fixtures), reconstructing actual results, evaluating metrics, and writing reports. The graph preserves deterministic metrics, RAGAS, and structured Gemini judging as distinct capabilities. Edges reflect the README and shown caller wiring; unsampled implementation details are not inferred.
Main entry points: run_evaluation.py Evaluation runner · golden_dataset.py Golden Dataset · trace_database.py Read-only DB engine · trace_repository.py Trace repository
Components 16
Run orchestration
- Evaluation runner
scripts/run_evaluation.py - Read-only DB engine
src/trace_database.py - Trace repository
src/trace_repository.py - Trace reconstructor
src/trace_reconstructor.py - Actual RAG result
src/trace_reconstructor.py
Evaluation inputs
- Evaluation config
config/evaluation.yaml - Golden Dataset
src/golden_dataset.py - Dataset records
datasets/golden/golden_dataset.json
Evaluation methods
- Deterministic metrics
src/evaluator.py - RAGAS metrics
src/ragas_evaluator.py - Gemini judge
src/gemini_judge.py
Reports and storage
- Report generator
src/reporting.py - Run reports
results
Other components
- Operator
- Trace database
- Gemini API
Connections 19
- Operator → Evaluation runner: runs
README.md - Evaluation runner → Evaluation config: uses
README.md - Evaluation runner → Golden Dataset: loads cases
scripts/run_evaluation.py - Golden Dataset → Evaluation config: reads settings
README.md - Golden Dataset → Dataset records: loads records
README.md - Evaluation runner → Read-only DB engine: creates engine
scripts/run_evaluation.py - Read-only DB engine → Trace database: connects read-only
README.md - Evaluation runner → Trace repository: fetches traces
scripts/run_evaluation.py - Trace repository → Trace database: selects trace rows
src/trace_repository.py - Evaluation runner → Trace reconstructor: reconstructs traces
scripts/run_evaluation.py - Trace reconstructor → Actual RAG result: produces
src/trace_reconstructor.py - Evaluation runner → Deterministic metrics: computes metrics
scripts/run_evaluation.py - Deterministic metrics → Actual RAG result: compares results
src/evaluator.py - Evaluation runner → RAGAS metrics: requests scores
scripts/run_evaluation.py - RAGAS metrics → Gemini API: calls model
README.md - Evaluation runner → Gemini judge: requests judgment
scripts/run_evaluation.py - Gemini judge → Gemini API: calls model
src/gemini_judge.py - Evaluation runner → Report generator: writes reports
scripts/run_evaluation.py - Report generator → Run reports: writes JSON CSV HTML
README.md