How ankitnayak-eth/epsteinfiles-rag works
Project type and purpose
• This is a full-stack Retrieval-Augmented Generation (RAG) pipeline built around the “Epstein Files 20K” dataset.
• It consists of a batch data‐ingestion pipeline (download → clean → chunk → embed), a real-time query component (FastAPI), and a Streamlit UI for interactive querying.
File structure
Top level:
• ingest/ – four standalone Python scripts to process raw documents into vector embeddings
• api/main.py – FastAPI application exposing a REST interface for querying
• app.py – Streamlit UI that consumes the REST API
• requirements.txt, .env.example – configuration & dependency management
Key points:
• The “ingest” folder is purely batch-oriented; each script is one stage in the ETL pipeline.
• The API and UI are separate processes communicating over HTTP.
• There is an external dependency on a vector database (Chroma), an LLM inference service (Groq), and a sentence transformer model.
README for additional insights
• Three stages:
Stage 1 (Data Prep): download, clean, chunk, embed → stored in Chroma DB
Stage 2 (Retrieval): FastAPI performs MMR search over embeddings
Stage 3 (Answer Generation): queried chunks + question → Groq-backed LLaMA → answer
• Tech stack: Python 3.11, FastAPI, Streamlit, LangChain, Chroma, Sentence Transformers, Groq API
• Runtime setup: two terminals (one runs uvicorn api.main:app, the other runs Streamlit)
a. Main components:
1. Data Source (Hugging Face dataset)
2. Ingestion Pipeline (four scripts)
3. Vector Store (Chroma DB)
4. Embedding Engine (Sentence Transformer)
5. REST API (FastAPI service)
6. Retrieval & MMR module (inside API)
7. LLM Inference Service (Groq Cloud for LLaMA)
8. Web UI (Streamlit app)
b. Relationships & interactions:
• At query time: Web UI → REST API → Retrieval module queries Chroma DB → top-k chunks returned → LLM Inference Service → final answer → REST API → Web UI
c. Architectural patterns & principles:
• Separation of concerns: batch ETL vs. real-time services
• Modular, script-based pipeline for data prep
• Microservice style between UI and API
• Asynchronous HTTP handling in FastAPI for low-latency queries
• Externalized configuration via environment variables
d. Technologies & frameworks:
• Python 3.11, Uvicorn + FastAPI, Streamlit, LangChain orchestration, Chroma vector store, Sentence Transformers for embeddings, Groq Cloud LLM inference