How ajai-r/comparative-rag-benchmark works
This repository is a research platform for comparing three RAG strategies, with document management, interactive querying/comparison, and benchmark evaluation. The user-facing path runs from the dashboard through the API and comparison bridge into a selected pipeline, then returns answers and retrieval traces. The pipelines differ in dense, hybrid, and adaptive-agent retrieval stages. README and excerpts support the principal flows; links inside unsampled backend and Pipeline 2 internals are omitted where caller evidence is unavailable.
Main entry points: agent.py Agent Orchestrator · PIPELINE 1/main.py Dense Orchestrator · pipeline_bridge.py Pipeline Bridge · PIPELINE 2/main.py Hybrid Orchestrator
Components 29
Web dashboard
- Dashboard App
frontend/src/App.jsx - Chat View
frontend/src/components/ChatView.jsx - Comparison View
frontend/src/components/ComparisonDashboard.jsx - Document Manager
frontend/src/components/DocumentManager.jsx - API Client
frontend/src/services/api.js
API service
- REST API
backend/server.py - Pipeline Bridge
backend/pipeline_bridge.py - Document Ingestion
backend/document_processor.py
Dense RAG
- Dense Orchestrator
PIPELINE 1/main.py - Document Loader
PIPELINE 1/document_loader.py - Chunking
PIPELINE 1/chunking.py - Embeddings
PIPELINE 1/embeddings.py - Vector Store
PIPELINE 1/vector_store.py - Dense Retrieval
PIPELINE 1/retrieval.py - Answer Generation
PIPELINE 1/generation.py
Hybrid RAG
- Hybrid Orchestrator
PIPELINE 2/main.py - BM25 Index
PIPELINE 2/bm25_index.py - Hybrid Retrieval
PIPELINE 2/retrieval.py - Rank Fusion
PIPELINE 2/fusion.py - Cross-Encoder Reranker
PIPELINE 2/reranker.py
Agentic RAG
- Agent Orchestrator
PIPELINE 3/agent.py - Query Classifier
PIPELINE 3/classifier.py - Query Decomposer
PIPELINE 3/decomposer.py - Math Tool
PIPELINE 3/calculator.py - Evidence Checker
PIPELINE 3/sufficiency.py
Benchmarking
- Benchmark Runner
Evaluation/run_benchmark.py - Evaluation Metrics
Evaluation/metrics.py
Other components
- User
- OpenRouter LLM
Connections 35
- User → Chat View: asks
README.md - User → Comparison View: compares
README.md - Dashboard App → Chat View: renders
frontend/src/App.jsx - Dashboard App → Comparison View: renders
frontend/src/App.jsx - Dashboard App → Document Manager: renders
frontend/src/App.jsx - Chat View → API Client: submits queries
frontend/src/components/ChatView.jsx - Comparison View → API Client: requests comparisons
frontend/src/components/ComparisonDashboard.jsx - Document Manager → API Client: manages documents
frontend/src/components/DocumentManager.jsx - API Client → REST API: calls REST API
frontend/src/services/api.js - REST API → Pipeline Bridge: delegates requests
backend/server.py - Pipeline Bridge → Document Ingestion: uses for ingestion
backend/pipeline_bridge.py - Pipeline Bridge → Dense Orchestrator: dispatches pipeline
- Pipeline Bridge → Hybrid Orchestrator: dispatches pipeline
- Pipeline Bridge → Agent Orchestrator: dispatches pipeline
- Dense Orchestrator → Document Loader: loads documents
PIPELINE 1/main.py - Dense Orchestrator → Chunking: creates chunks
PIPELINE 1/main.py - Dense Orchestrator → Embeddings: embeds chunks
PIPELINE 1/main.py - Dense Orchestrator → Vector Store: stores vectors
PIPELINE 1/main.py - Dense Orchestrator → Dense Retrieval: retrieves matches
PIPELINE 1/main.py - Dense Orchestrator → Answer Generation: generates answer
PIPELINE 1/main.py - Answer Generation → OpenRouter LLM: requests completion
PIPELINE 1/generation.py - Hybrid Orchestrator → BM25 Index: loads sparse index
PIPELINE 2/main.py - Hybrid Orchestrator → Hybrid Retrieval: runs retrieval
PIPELINE 2/main.py - Hybrid Retrieval → Rank Fusion: fuses ranks
PIPELINE 2/retrieval.py - Hybrid Orchestrator → Cross-Encoder Reranker: initializes reranker
PIPELINE 2/main.py - Hybrid Orchestrator → Answer Generation: generates answer
PIPELINE 2/main.py - Agent Orchestrator → Query Classifier: classifies query
PIPELINE 3/agent.py - Agent Orchestrator → Query Decomposer: decomposes queries
PIPELINE 3/agent.py - Agent Orchestrator → Math Tool: uses math tool
PIPELINE 3/agent.py - Agent Orchestrator → Evidence Checker: checks evidence
PIPELINE 3/agent.py - Agent Orchestrator → Dense Retrieval: retrieves dense evidence
- Agent Orchestrator → Hybrid Retrieval: retrieves hybrid evidence
- Agent Orchestrator → Answer Generation: generates answer
PIPELINE 3/main.py - Benchmark Runner → Evaluation Metrics: computes scores
Evaluation/run_benchmark.py - Benchmark Runner → Agent Orchestrator: evaluates agent
Evaluation/run_benchmark.py