How nguyenhongsac/rag_chatbot works
Project Type and Purpose
• This is a Python-based full-stack web application implementing a Retrieval-Augmented Generation (RAG) chatbot.
• The backend uses Flask (app.py) to expose HTTP endpoints; the front-end is a simple static HTML/JS/CSS client under the “Simple_test” folder.
• Core functionality: index a document collection, build or load a vector/graph store, retrieve relevant passages, call an external LLM API, and return generated responses.
File Structure Highlights
• Top-level scripts:
• app.py – Flask web server
• main.py – orchestrator or CLI entry point
• indexing_knowledge.py – batch job to ingest and index documents
• llm_test.py – standalone LLM invocation tests
• “components” directory: modular pipeline pieces
• graph.py – knowledge-graph builder/manager
• indexing_pipeline.py – ingestion → embedding → store
• rag_pipeline.py – retrieval + generation orchestration
• llm_model.py – wrapper around external LLM service
• Front-end sample (“Simple_test”): static assets (HTML, CSS, JS) demonstrating how the client consumes the Flask API.
Key System Components
a. User’s Browser / Front-end Client
b. Flask Backend (app.py) with REST/WebSocket endpoints
c. RAG Pipeline Modules (components/indexing_pipeline, components/rag_pipeline, components/llm_model)
d. Knowledge Store
– Vector database (e.g., FAISS, Pinecone) or local embedding store
– Graph database or in-memory graph from components/graph.py
e. Batch Indexing Service (indexing_knowledge.py)
f. External LLM Service (OpenAI API or similar)
g. Document Source (static files, external data feeds)
Relationships and Data Flow
• Separate flow: Batch Indexer (indexing_knowledge.py) → Indexing Pipeline → Knowledge Store
Architectural Patterns and Principles
• Layered/pipeline architecture: ingestion → storage → retrieval → generation → API
• Separation of concerns: distinct modules for indexing, retrieval, LLM interfacing
• Stateless web tier (Flask) vs. stateful data store (vector or graph DB)