How sridharmalladi/raglens works
1. What this repo is
- RAGLens is a live retrieval-augmented generation benchmarking app.
- Its main purpose is to run the same query across four retrieval configurations and compare answers and LLM-as-judge scores side by side:
- No RAG
- Dense RAG
- Hybrid RAG
- Hybrid + reranking
2. Top-level runtime shape
- The app is served from main.py, with the README showing it is launched with Uvicorn on port 7860.
- The repo is packaged for Docker / HF Spaces, so the deployed unit is a single container exposing both backend logic and the web UI.
3. Frontend boundary
- static/index.html, static/app.js, and static/styles.css are the client-side presentation layer.
- This front end is likely thin: it renders the comparison dashboard and calls backend endpoints for generation, retrieval, and monitoring data.
- The architecture boundary is clear: static UI in static/, Python orchestration and ML logic in src/.
4. Backend orchestration layer
- src/ contains the core application subsystems:
- inference.py for LLM calls and answer generation
- retrieval.py for the retrieval strategies
- evaluation.py for scoring/benchmarking
- monitoring.py for time-based stats and dashboard data
- scheduler.py for periodic jobs
- storage.py for persistence
- corpus.py for corpus loading/chunk handling
- models.py for shared data structures/config objects
- This looks like a service-oriented Python backend, where the request path composes retrieval, generation, evaluation, and storage.
5. Retrieval subsystem
- The README defines the retrieval stack as:
- Dense retrieval using BGE embeddings and FAISS
- Sparse retrieval using BM25
- Hybrid retrieval combining both
- Reranking using a BGE cross-encoder
- The main architectural distinction is that the retrieval layer is not a single index; it is a strategy switchboard that can execute multiple pipelines for comparison.
6. Generation + judge evaluation
- Answer generation uses Groq llama-3.1-8b-instant.
- Evaluation is separate from generation and uses LLM-as-judge scoring for:
- faithfulness
- relevancy
- precision
- This means the system has two LLM-facing flows:
- one for producing candidate answers
- one for scoring those answers against the retrieved context and/or query
7. Corpus and indexing pipeline
- The corpus is stored under corpus/raw and corpus/processed, with corpus/processed/chunks.json as a persisted processed artifact.
- build_corpus.ipynb suggests offline preprocessing/index-building happens separately from live query serving.
- src/corpus.py likely loads and normalizes these chunks for retrieval, keeping the serving path independent of raw document ingestion.
8. Persistence and state
- src/storage.py and the README indicate SQLite is used for persistence.
- That storage likely holds benchmark runs, scores, monitoring history, or cached results.
- The repo also includes adapters/, which is probably intended as an integration boundary for external data sources or future ingestion connectors, though it is currently empty.
9. Scheduling and monitoring
- src/scheduler.py and the README’s mention of APScheduler hourly cron indicate background jobs are part of the system.
- src/monitoring.py plus the “7-day monitoring dashboard” suggest the app collects and aggregates run history over time, separate from live query execution.
- This creates a second major data flow: periodic benchmark collection → SQLite → monitoring dashboard, independent of direct user queries.
10. Configuration and secrets
- config.py and .env.example show the app is environment-driven.
- The key external dependency is GROQ_API_KEY, which gates both generation and judge evaluation.
- This keeps cloud API access out of the code path and makes deployment portable across local dev and container hosting.
11. Testing and quality boundary
- tests/test_retrieval.py, tests/test_generation.py, and tests/test_evaluation.py map directly to the three core ML subsystems.
- This is useful for architecture diagrams because the repo’s test coverage mirrors the runtime decomposition: retrieval, generation, and evaluation are treated as separate units.
12. End-to-end data flow
- A user query enters the backend through the web UI.
- The backend runs the query through one or more retrieval strategies.
- Retrieved chunks are fed to Groq for answer generation.
- The generated output is scored by the judge model.
- Results are persisted in SQLite and surfaced in the comparison/monitoring UI.
- Scheduled jobs keep the monitoring data fresh without requiring user interaction.
13. Key architectural takeaway
- This is not just a demo app; it is a benchmarking system with a clean separation between:
- offline corpus preparation
- online retrieval/generation
- evaluation
- persistence/monitoring
- static presentation
- The most important diagram boundary is between the query-time RAG pipeline and the scheduled benchmarking/analytics pipeline, both sharing the same storage and corpus artifacts.
Main entry points: main.py Main app · retrieval.py Retrieval · inference.py Generation · evaluation.py Judge
Components 18
UI
Static presentation layer for the comparison dashboard.
- Dashboard · Primary web page for comparing RAG configurations.
static/index.html - Client logic · Front-end behavior and API calls for the dashboard.
static/app.js - Styles · Visual presentation for the dashboard.
static/styles.css
Backend
Python app, orchestration, retrieval, evaluation, and persistence.
- Main app · Serves the app and routes requests into backend orchestration.
main.py - Config · Environment-driven settings and secrets.
config.py - Models · Common data structures and configuration objects.
src/models.py - Retrieval · Coordinates dense, sparse, hybrid, and reranked retrieval.
src/retrieval.py - Corpus · Loads and normalizes processed corpus chunks for serving.
src/corpus.py - Generation · Builds prompts and produces candidate answers from retrieved context.
src/inference.py - Judge · Scores answers with faithfulness, relevancy, and precision metrics.
src/evaluation.py - Monitoring · Aggregates run history and dashboard stats over time.
src/monitoring.py
Data
Corpus artifacts and persisted state used by the app.
- SQLite · Stores runs, scores, cached results, and monitoring history.
src/storage.py - Chunks · Persisted processed corpus artifact used by retrieval.
corpus/processed/chunks.json - Raw corpus · Original corpus inputs for offline preprocessing.
corpus/raw
Ops
Containerization and scheduled/background execution.
- Scheduler · Runs periodic benchmark and monitoring jobs.
src/scheduler.py - Corpus build · Offline preprocessing and index-building workflow.
build_corpus.ipynb - Adapters · Placeholder boundary for external ingestion or integration connectors.
adapters - Container · Containerized deployment for local and hosted execution.
Dockerfile
Connections 22
- Main app → Dashboard: serves · Main app exposes the dashboard UI.
- Main app → Client logic: loads · Browser client behavior is delivered with the app.
- Main app → Retrieval: query path · Requests flow into retrieval strategy execution.
- Main app → Generation: generates · Backend calls the LLM answer generation flow.
- Main app → Judge: scores · Backend runs judge scoring for benchmark outputs.
- Main app → Monitoring: shows · Dashboard surfaces monitoring and history data.
- Client logic → Main app: API calls · Front end invokes backend endpoints.
- Retrieval → Corpus: uses · Retrieval operates over processed chunks.
- Retrieval → Chunks: reads · Chunk artifact is the persisted retrieval input.
- Retrieval → Models: shared types · Uses common request and result schemas.
- Retrieval → SQLite: stores results · Persists benchmark outcomes and cached run data.
- Generation → Config: uses keys · LLM access is driven by environment config.
- Generation → Models: shared types · Consumes and returns structured payloads.
- Judge → Generation: evaluates output · Judge scores generated answers.
- Judge → SQLite: records scores · Persists evaluation outcomes.
- Monitoring → SQLite: aggregates · Monitoring dashboard reads persisted history.
- Scheduler → Retrieval: periodic runs · Background jobs trigger benchmark executions.
- Scheduler → Monitoring: refreshes · Scheduled jobs keep metrics current.
- Corpus build → Raw corpus: processes · Offline notebook transforms source documents.
- Corpus build → Chunks: produces · Offline preprocessing emits processed chunks.
- Container → Main app: packages · Container image bundles the runtime entrypoint.
- Adapters → Raw corpus: ingests · Integration boundary for future data sources.