How prodip1023/rag-application-1 works
Project Type and Purpose
• This is an open‐source Retrieval‐Augmented Generation (RAG) tool.
• Purpose: ingest a text corpus, index it with FAISS, and at runtime retrieve relevant chunks to augment prompts sent to an LLM (e.g., OpenAI) for generation.
Main Architectural Components
a. Data Ingestion & Indexing (Offline)
• Source: data/mydata.txt
• Chunking module: utils/chunking.py
• Embedding module: utils/embedding.py
• FAISS store: faiss_store/index.faiss + chunk_mapping.pkl
b. Retrieval & Prompt Construction (Online)
• Retrieval module: utils/retrieval.py
• Prompt builder: utils/prompt.py
c. Completion (LLM)
• Completion module: utils/completion.py calls external OpenAI API
d. Orchestration & API Layer
• app.py: coordinates user queries, retrieval, prompt assembly, completion, and response
Relationships & Data Flows
• Offline Flow:
1) Read raw text → 2) Chunker → 3) Embed → 4) FAISS index build → Persist index & mapping
• Online Flow:
1) User request → app.py → 2) Embed query (optional) → 3) FAISS retrieval → 4) Prompt build → 5) Send to OpenAI → 6) Return generation to user
Architectural Patterns & Design Principles
• Modular pipeline: each utility (chunking, embedding, retrieval, prompt, completion) is a separate component
• Separation of concerns: offline indexing vs. online serving
• Single‐process/script based orchestration (no microservices)
• External service dependency: OpenAI API
Technologies & Frameworks to Call Out
• Python 3.10 (specified in environment)
• FAISS (vector store)
• OpenAI Python SDK (in utils/completion.py)
• Conda / pip for environment & dependencies
Essential Elements
• Clear component boundaries (Offline vs. Online)
• All five utils modules as separate blocks
• The FAISS store and mapping file as persistent storage
• The external OpenAI API
• The orchestration script app.py
• Separates indexing & serving phases
• Clearly labels modules and data flows
• Emphasizes modularity and extension points inherent in this RAG application