How athina-ai/rag-cookbooks works
Main Components
• Data Ingestion & Processing: Start with the “data” component where documents (like the tesla_q3.pdf and context.csv) are stored. This is the source for creating document chunks and their embeddings.
• Indexing (Embeddings Generation & Storage): Show a component where documents are split into chunks, embeddings are generated, and these embeddings are stored in a vector store. Technologies mentioned here include FAISS, Chromadb, Pinecone, Weaviate, etc.
• Retrieval Module (Retriever): Represent the module that picks the most relevant documents based on a query using vector similarity. This is a core part of each notebook’s implementation.
• Query Augmentation: Illustrate the “augment” part where the original user query is combined with the retrieved context to form a complete prompt for the language model.
• Generation (LLM): Identify the component where the LLM (typically working with LangChain and possibly LangSmith) generates the final answer based on the augmented query.
• Evaluation & Feedback: From the README, it’s clear that RAG evaluation is also a focus. Represent a feedback loop or evaluation component (including metrics on accuracy and relevance) used to fine-tune or gauge performance.
• Variants & Techniques: Since the repository includes various notebooks in folders like advanced_rag_techniques, agent_techniques, and agentic_rag_techniques, it is good to represent these as specialized modules or variant pipelines that might alter or extend the core pipeline:
• Advanced Techniques (e.g., Hybrid RAG, Contextual RAG, Fusion RAG)
• Agent-based Techniques (e.g., Basic Agentic RAG, Corrective RAG, Self RAG, Adaptive RAG, ReAct RAG)
Depicting Relationships and Data Flow
• How raw files are processed into document chunks.
• How embeddings flow into the vector store.
• How a user query is applied: first being processed by the Retriever to fetch relevant document chunks.
• How the retrieved data is then merged with the query (Augment stage) before being passed to the Language Model.
• Optionally, include a feedback loop from Generation back to Evaluation.
• For the advanced and agentic variations, you could indicate branching: show how the common pipeline is adapted or extended for each technique. For instance, “Fusion RAG” might add a component for sub-query generation or document ranking (i.e., Reciprocal Rank Fusion), whereas “Self RAG” includes a self-reflection or verification module.
Architectural Patterns and Tools
• Component with the relevant tools (e.g., LangChain for orchestration, FAISS/Pinecone for vector search, Chromadb/Weaviate for storage, etc.).
• “agent” based modifications in the agentic techniques, where autonomous modules (or agents) might dynamically adjust retrieval methods or perform corrective actions.