How campusx-official/rag-eval-deepeval works
This repository is a transcript-grounded RAG course assistant accompanied by a broad evaluation toolkit. The user workflow is chat question → retrieval and reranking → answer generation, with retrieved context shown alongside the answer. A separate evaluation workflow runs quality, safety, and operational checks against golden datasets or live pipeline output. The README is empty, and several evaluation and pipeline modules were not sampled; their presence is clear from the tree, but uncertain internal wiring is omitted.
Main entry points: run_suite.py Suite runner · eval_rag_pipeline.py RAG quality · eval_retriever.py Retrieval quality · rag_pipeline.py RAG pipeline
Components 20
Course assistant
- Chat UI
src/app.py - RAG pipeline
src/rag_pipeline.py - Answer generator
src/generator.py
Knowledge retrieval
- Reranking retriever
src/reranker.py - Vector retriever
src/retriever.py - Course transcripts
data - Chroma store
src/retriever.py
Evaluation suite
- Suite runner
evals/run_suite.py - Eval harness
evals/harness.py - RAG quality
evals/eval_rag_pipeline.py - Retrieval quality
evals/eval_retriever.py - Reranker quality
evals/eval_retriever_with_reranker.py - Safety checks
evals - Operational checks
evals/eval_ops.py - Other evals
evals
Evaluation data
- Golden datasets
goldens
Other components
- Chat user
- LLM service
- LLM judge
- Embedding service
Connections 27
- Chat user → Chat UI: asks
- Chat UI → RAG pipeline: invokes
- RAG pipeline → Reranking retriever: retrieves
- RAG pipeline → Answer generator: generates
- Reranking retriever → Vector retriever: fetches candidates
- Vector retriever → Chroma store: searches
- Course transcripts → Vector retriever: loads and chunks
- Vector retriever → Embedding service: embeds queries
- Answer generator → LLM service: requests completion
- Suite runner → RAG quality: dispatches
- Suite runner → Retrieval quality: dispatches
- Suite runner → Reranker quality: dispatches
- Suite runner → Safety checks: dispatches
- Suite runner → Operational checks: dispatches
- Suite runner → Other evals: dispatches
- RAG quality → RAG pipeline: evaluates
- Retrieval quality → Reranking retriever: evaluates
- Reranker quality → Reranking retriever: evaluates
- RAG quality → Golden datasets: loads queries
- Retrieval quality → Golden datasets: loads cases
- Reranker quality → Golden datasets: loads cases
- RAG quality → Eval harness: summarizes
- Retrieval quality → Eval harness: summarizes
- RAG quality → LLM judge: scores outputs
- Retrieval quality → LLM judge: scores retrieval
- Reranker quality → LLM judge: scores retrieval
- Operational checks → RAG pipeline: measures