How chintamanil/review-genie works
Project type and purpose
• A monolithic Python application implementing a Retrieval-Augmented Generation (RAG) evaluation pipeline for Amazon-product queries.
• Purpose: retrieve relevant docs via FAISS, generate answers with an LLM, evaluate those answers across four metrics, and log everything to LangSmith.
File structure
• app.py: the single orchestrator (retrieval → generation → evaluation → logging).
• faiss_index/: pre-built FAISS vector store files (index.faiss, index.pkl).
• prod_small.csv: evaluation dataset (questions + ground truths).
• README.md: documentation and setup instructions.
README for architecture hints
• Uses LangChain for pipeline construction.
• OpenAI embeddings for retrieval, OpenAI chat for generation and evaluation.
• FAISS for nearest-neighbor search over product docs.
• LangSmith for tracing and experiment management.
• Four separate evaluators (correctness, relevance, groundedness, retrieval relevance).
Main components & interactions
a. Data Layer
• Evaluation Dataset (prod_small.csv)
• FAISS Vector Store (faiss_index/)
b. Retrieval Module
• OpenAI Embedding API → query embeddings
• FAISS index lookup → retrieved docs
c. Generation Module
• OpenAI Chat API → generates answer from retrieved docs
d. Evaluation Module
• Four LLM-based evaluators (each calls OpenAI to judge one metric)
e. Orchestration & Tracing
• app.py ties all steps
• LangSmith SDK logs traces & results
Data flow & relationships
User Query → Orchestrator (app.py)
→ Embedding Service (OpenAI Embeddings)
→ FAISS Index lookup
→ Retrieved Documents → LLM Generation (OpenAI Chat)
→ Generated Answer → Evaluation Module (4 evaluator calls)
→ Results & traces → LangSmith Dashboard
Architectural patterns & design principles
• Pipeline pattern: sequential stages (retrieve → generate → evaluate).
• Single Responsibility: each evaluator addresses one quality metric.
• Externalized state: FAISS index on disk; LangSmith stores experiment metadata.
• Clear separation of concerns across modules inside one script.
Key technologies
• Python 3.8+
• LangChain (pipeline helpers)
• FAISS (vector search)
• OpenAI API (embeddings + chat)
• LangSmith SDK (observability & experiment tracking)
Essential details not to omit
• The FAISS index files as the core retrieval mechanism.
• The four distinct evaluators and their LLM calls.
• Environment variables (API keys, tracing flags) near the orchestrator.