How brainscorp/-rag_llm_chatbot works
Project type and purpose
• This is a full-stack, single-service Python application built with Streamlit that implements a Retrieval-Augmented Generation (RAG) pipeline using OpenAI’s GPT-4 model, LangChain abstractions, ChromaDB and FAISS for vector search.
• Its core use case is answering domain-specific questions by retrieving relevant documents from a custom knowledge base, then generating contextually grounded answers via an LLM.
File structure
• Top level:
• app.py – Streamlit entry point (UI + orchestrator)
• data/ – raw documents for ingestion
• model/llm.py – wrapper around the LLM (OpenAI GPT-4)
• utils/ – modular logic (data2vec, retriever, reasoning_query)
• vector_db/ – persistent vector store artifacts (Chroma SQLite, FAISS index, Pickle)
• requirements.txt, README.md, .gitignore
• Patterns & configuration:
• “utils” implements the retrieval and preprocessing pipeline (vector embedding, metadata filtering)
• “vector_db” holds on-disk storage for the vector store
• No microservices – everything runs in one Python process, orchestrated by Streamlit in app.py
README for additional insights
• Confirms RAG architecture: Retriever + Generator + Orchestrator
• Dependencies: LangChain, ChromaDB, FAISS, Streamlit, OpenAI SDK
• Mentions data ingestion script (data2vec.py) and custom retriever extension points (utils/retriever.py)
Diagramming recommendations
• Data ingestion: Data folder → Embedding Generator → Vector Store
• Query path: User → UI → Orchestrator → Retriever → Vector Store → Generator → UI
• UI layer (e.g., blue)
• Orchestration & services (e.g., green)
• Data stores (e.g., orange)
• External API (e.g., red)
• Clouds for external services (OpenAI)
• Rectangles for modules/services