How prasad-j155/rag-automated-evaluation-pipeline works
Project type and purpose
• This is a full-stack Python application combining a batch pipeline and a web dashboard.
• Purpose: build a Hybrid Retrieval-Augmented Generation (RAG) system that
• Ingests and indexes web content (fixed + dynamic URLs)
• Performs sparse (BM25) and dense (FAISS) retrieval
• Fuses results via Reciprocal Rank Fusion (RRF)
• Generates answers using an LLM (google/flan-t5-base)
• Runs an automated evaluation suite (MRR, BERTScore)
• Exposes results via Streamlit UI
File structure
• Top-level scripts:
• run_pipeline.py – end-to-end orchestration of data gen, indexing, question gen, retrieval, generation, evaluation, report export
• app.py – Streamlit front end for interactive querying and visualization
• src/ modules:
• config.py – centralize all paths, hyperparameters (RRF k-value, index names, report output folder)
• data_loader.py – fetch fixed_urls.json + Wikipedia URLs, build corpus
• rag_engine.py – wrap sparse search (BM25Okapi), dense search (all-MiniLM-L6-v2 + FAISS), run RRF fusion
• evaluation.py – question generation, run ablation studies (Dense vs Sparse vs Hybrid), compute MRR and BERTScore, save charts/CSVs
• Requirements.txt – Python dependencies (faiss-cpu, rank-bm25, sentence-transformers, transformers, streamlit, bert_score)
Extract architecture notes from the README
• Data flows from “Data Source” → “Retrieval Engine” → “LLM Generator” → “Evaluation”
• Two retrieval subcomponents (sparse vs dense) feed into a fusion layer
• A single command (run_pipeline.py) drives the end-to-end flow
• app.py gives a live dashboard to query the same engine and view saved reports
a. Main components:
1. Data Sources
– Fixed URLs (local JSON)
– Dynamic URLs (Wikipedia API)
2. Data Loader
3. Indexing layer
– Sparse index (BM25Okapi)
– Dense index (Sentence-Transformer embeddings + FAISS)
4. Retrieval Engine
– BM25 search
– FAISS search
5. Fusion Module (Reciprocal Rank Fusion)
6. LLM Generator (google/flan-t5-base)
7. Evaluation Suite
– Question generator
– Metric calculators (MRR, BERTScore)
– Report exporter (charts + CSV into reports/)
8. Web Dashboard (Streamlit)
• Data Sources → Data Loader → Indexing layer
• Indexing layer → Retrieval Engine (two parallel branches: sparse and dense)
• Both retrieval outputs → Fusion Module → merged result
• Merged result + context → LLM Generator → answers
• Answers + ground-truth questions → Evaluation Suite → metrics + reports
• Web Dashboard queries Retrieval Engine / Fusion / Generator on demand
• Reports folder is a storage artifact
c. Architectural patterns and design principles:
– Modular service layers: ingestion, indexing, retrieval, fusion, generation, evaluation, presentation
– Pipeline orchestration script (run_pipeline.py) implements a linear DAG of tasks
– Single monolithic codebase with clear separation by Python module
d. Technologies/frameworks:
– BM25Okapi from rank-bm25
– SentenceTransformers for embedding generation
– FAISS vector store for fast nearest-neighbor search
– Reciprocal Rank Fusion (RRF) algorithm k=60
– Transformers LLM (google/flan-t5-base) for generation
– bert_score library for BERTScore
– Streamlit for UI dashboard
– Python 3.8+