How apoorvvats181/genai-research-tool works
Project Type and Purpose
• This is an open‑source, Python‑based research assistant CLI/tool (with a companion notebook) rather than a full‑stack web application.
• Purpose: Ingest multiple web article URLs, chunk and preprocess text via LangChain document loaders, generate embeddings with OpenAI, store/retrieve with FAISS, and answer user queries with source‑aware responses.
File Structure
• Top level:
• main.py – the orchestrator script and CLI entry point
• notebooks/retrieval.ipynb – interactive demo of ingestion/retrieval flows
• requirements.txt – lists LangChain, FAISS, OpenAI SDK, etc.
• No frontend folder—this is purely backend/CLI code + notebook.
• No microservices or network modules—everything runs locally.
README Insights
• Core components mentioned:
• LangChain document loaders (chunking/preprocessing)
• OpenAI Embeddings
• FAISS vector store
• Local JSON/vector storage
• Use cases: ingest, index, and query research articles.
Main Components & Their Relationships
a. User Interface
• CLI (main.py) and optional Jupyter Notebook for interactive runs.
b. Ingestion & Preprocessing
• Document Loader Module (LangChain) fetches and chunks URLs.
c. Embedding Service
• OpenAI Embeddings API called via LangChain.
d. Vector Store
• FAISS local index for similarity search.
e. Knowledge Base Storage
• JSON files & persistent FAISS index on disk.
f. Retrieval & Response Generation
• Query module: retrieves top‑k docs via FAISS → constructs prompt → calls OpenAI Chat endpoint → returns answers with citations.
Architectural Patterns and Design Principles
• Pipeline pattern: sequential ingestion → embed → index → query → respond.
• Separation of concerns: distinct modules for loading, embedding, storing, querying, and prompting.
• Local persistence: offline vector store + JSON ensures repeatable runs without re‑ingestion.