# athina-ai/rag-cookbooks architecture

> Architecture diagram of the GitHub repository athina-ai/rag-cookbooks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/athina-ai/rag-cookbooks
- Repository: https://github.com/athina-ai/rag-cookbooks

## Overview

##### Main Components
- Data Ingestion & Processing: Start with the “data” component where documents (like the tesla_q3.pdf and context.csv) are stored. This is the source for creating document chunks and their embeddings.
- Indexing (Embeddings Generation & Storage): Show a component where documents are split into chunks, embeddings are generated, and these embeddings are stored in a vector store. Technologies mentioned here include FAISS, Chromadb, Pinecone, Weaviate, etc.
- Retrieval Module (Retriever): Represent the module that picks the most relevant documents based on a query using vector similarity. This is a core part of each notebook’s implementation.
- Query Augmentation: Illustrate the “augment” part where the original user query is combined with the retrieved context to form a complete prompt for the language model.
- Generation (LLM): Identify the component where the LLM (typically working with LangChain and possibly LangSmith) generates the final answer based on the augmented query.
- Evaluation & Feedback: From the README, it’s clear that RAG evaluation is also a focus. Represent a feedback loop or evaluation component (including metrics on accuracy and relevance) used to fine-tune or gauge performance.
- Variants & Techniques: Since the repository includes various notebooks in folders like advanced_rag_techniques, agent_techniques, and agentic_rag_techniques, it is good to represent these as specialized modules or variant pipelines that might alter or extend the core pipeline:
- Advanced Techniques (e.g., Hybrid RAG, Contextual RAG, Fusion RAG)
- Agent-based Techniques (e.g., Basic Agentic RAG, Corrective RAG, Self RAG, Adaptive RAG, ReAct RAG)

##### Depicting Relationships and Data Flow
- How raw files are processed into document chunks.
- How embeddings flow into the vector store.
- How a user query is applied: first being processed by the Retriever to fetch relevant document chunks.
- How the retrieved data is then merged with the query (Augment stage) before being passed to the Language Model.
- Optionally, include a feedback loop from Generation back to Evaluation.
- For the advanced and agentic variations, you could indicate branching: show how the common pipeline is adapted or extended for each technique. For instance, “Fusion RAG” might add a component for sub-query generation or document ranking (i.e., Reciprocal Rank Fusion), whereas “Self RAG” includes a self-reflection or verification module.

##### Architectural Patterns and Tools
- Component with the relevant tools (e.g., LangChain for orchestration, FAISS/Pinecone for vector search, Chromadb/Weaviate for storage, etc.).
- “agent” based modifications in the agentic techniques, where autonomous modules (or agents) might dynamically adjust retrieval methods or perform corrective actions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Pipeline
    DataSource["Data Source (PDF, CSV)"]:::data
    Indexing["Indexing & Embedding (Document Processing, Embeddings, Vector Store)"]:::indexing
    Retrieval["Retriever (Vector Similarity Search)"]:::retrieval
    QueryAugmentation["Query Augmentation (Rewrite/Combine Query)"]:::augmentation
    LLM["Language Model (LLM Generation)"]:::generation
    Evaluation["Evaluation & Feedback (Performance Metrics)"]:::evaluation

    %% Variant Techniques
    VariantAdvanced["Advanced Techniques (Hybrid, Contextual, Fusion, Naive)"]:::variantAdvanced
    VariantAgent["Agent-based Techniques (React, Reflexion, Rewoo)"]:::variantAgent
    VariantAgentic["Agentic Approaches (Adaptive, Corrective, Self)"]:::variantAgentic

    %% Connections for core pipeline
    DataSource -->|"process"| Indexing
    Indexing -->|"generate embeddings"| Retrieval
    Retrieval -->|"default"| QueryAugmentation
    QueryAugmentation -->|"build prompt"| LLM
    LLM -->|"result"| Evaluation
    Evaluation -.->|"feedback"| Retrieval

    %% Connections for variant branches (branching from retrieval and converging into query augmentation)
    Retrieval -->|"advanced"| VariantAdvanced
    Retrieval -->|"agent-based"| VariantAgent
    Retrieval -->|"agentic"| VariantAgentic

    VariantAdvanced -->|"modify"| QueryAugmentation
    VariantAgent -->|"modify"| QueryAugmentation
    VariantAgentic -->|"modify"| QueryAugmentation

    %% Click Events
    click DataSource "https://github.com/athina-ai/rag-cookbooks/tree/main/data/"
    click Indexing "https://github.com/athina-ai/rag-cookbooks/tree/main/advanced_rag_techniques/"
    click Retrieval "https://github.com/athina-ai/rag-cookbooks/blob/main/advanced_rag_techniques/parent_document_retriever.ipynb"
    click QueryAugmentation "https://github.com/athina-ai/rag-cookbooks/blob/main/advanced_rag_techniques/rewrite_retrieve_read.ipynb"
    click LLM "https://github.com/athina-ai/rag-cookbooks/blob/main/advanced_rag_techniques/basic_unstructured_rag.ipynb"
    click Evaluation "https://github.com/athina-ai/rag-cookbooks/tree/main/agent_techniques/"
    click VariantAdvanced "https://github.com/athina-ai/rag-cookbooks/tree/main/advanced_rag_techniques/"
    click VariantAgent "https://github.com/athina-ai/rag-cookbooks/tree/main/agent_techniques/"
    click VariantAgentic "https://github.com/athina-ai/rag-cookbooks/tree/main/agentic_rag_techniques/"

    %% Styles
    classDef data fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef indexing fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef retrieval fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef augmentation fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef generation fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#D3D3D3,stroke:#333,stroke-width:2px;
    classDef variantAdvanced fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef variantAgent fill:#98FB98,stroke:#333,stroke-width:2px;
    classDef variantAgentic fill:#FFB6C1,stroke:#333,stroke-width:2px;
```
