# varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207 architecture

> Architecture diagram of the GitHub repository varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207
- Repository: https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207

## Overview

##### Project type and purpose
- This is an educational code repository accompanying the “Hands-On AI: RAG using LlamaIndex” LinkedIn Learning course.
- Its purpose is to guide learners through the concepts and code needed to build, evaluate, and extend Retrieval-Augmented Generation (RAG) systems using the LlamaIndex framework.

##### File structure
- Top-level directories “01_Getting_Started” through “07_Modular_RAG” each contain notebooks or markdown files illustrating a specific stage in the RAG pipeline: environment setup, fundamentals, ingestion, querying, evaluation, advanced pre-/post-retrieval, and modular RAG.
- The “helpers” package contains reusable Python modules for prompts, text cleaning, and utilities.
- Configuration files (.devcontainer, .env_example), CI workflows (.github/workflows/main.yml), and requirements.txt support reproducible environments and continuous integration.

##### README for additional insights
- The README describes the course scope: fundamentals of RAG, LlamaIndex integration, end-to-end pipelines, and critical evaluation.

##### Main components
a. Data Sources
- Raw data files (PDFs, text corpora) used in “01_Getting_Started” notebooks
b. Pre-processing
- helpers/text_cleaning_helpers.py
- helpers/utils.py
c. Ingestion Pipeline
- LlamaIndex loaders (02_Fundamental_Concepts_in_LlamaIndex/02_03_Loading_Data.ipynb)
- Indexing (02_04_Indexing.ipynb)
- Storage backends (vector DB of choice, configured in .env_example)
d. Query Pipeline
- Retrieval engines (02_06_Querying.ipynb, 03_Introduction_to_RAG/03_03_Query_Pipeline.ipynb)
- Prompt engineering (03_04_Prompt_Engineering_for_RAG.md)
e. Post-Retrieval Enhancements
- Node re-ranking, prompt compression, self-correcting (06_Advanced_RAG_Post_Retrieval)
f. Modular RAG Extensions
- Hybrid retrievers, agentic RAG, ensembles (07_Modular_RAG)
g. Evaluation
- Metrics and evaluation set creation (04_RAG_Evaluation)
h. Orchestration & Environment
- Jupyter notebooks as choreographers
- .devcontainer for environment, CI workflows for testing

##### Relationships and data flow
1. Raw Data → Pre-processing → Ingestion Pipeline → Vector Store
2. User Query → Query Pipeline → Retrieved Documents
3. Retrieved Documents → Post-Retrieval Modules → Final Prompt
4. Final Prompt → LLM → Model Response
5. Model Response + Ground Truth → Evaluation Module

##### Architectural patterns and design principles
- Pipeline pattern: clear separation of ingestion, retrieval, post-processing, and evaluation stages.
- Modular design: each stage is implemented in its own directory/notebook, helpers package centralizes reusable code.
- Pluggable backends: vector database and LLM providers are configurable via environment variables.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion & Storage
    subgraph "Data Ingestion & Storage"
        direction TB
        DataSources["Data Sources\n(01_07_Downloading_Our_Data.ipynb)"]:::dataBlue
        subgraph "Pre-processing"
            direction TB
            TextCleaning["Text Cleaning\nhelpers/text_cleaning_helpers.py"]:::ingestionBlue
            Utils["Utilities\nhelpers/utils.py"]:::ingestionBlue
        end
        subgraph "Ingestion Pipeline"
            direction TB
            Loading["Loading Data\n(02_03_Loading_Data.ipynb)"]:::ingestionBlue
            Indexing["Indexing\n(02_04_Indexing.ipynb)"]:::ingestionBlue
            Storing["Storing\n(02_05_Storing.ipynb)"]:::ingestionBlue
            IngestionOverview["Ingestion Overview\n(03_02_Ingestion_Pipeline.ipynb)"]:::ingestionBlue
        end
        VectorDB["Vector Store\n(pluggable backend)"]:::storage
    end

    %% Query & Retrieval
    subgraph "Query & Retrieval"
        direction TB
        UserQuery["User Query\n(via notebooks)"]:::external
        subgraph "Query Pipeline"
            direction TB
            QueryFund["Querying\n(02_06_Querying.ipynb)"]:::retrievalGreen
            QueryIntro["Query Pipeline\n(03_03_Query_Pipeline.ipynb)"]:::retrievalGreen
            PromptEng["Prompt Engineering\n(03_04_Prompt_Engineering_for_RAG.ipynb)"]:::retrievalGreen
        end
    end

    %% Post-Retrieval Enhancements
    subgraph "Post-Retrieval Enhancements" 
        direction TB
        NodeProc["Node Post-Processing\n(06_01_Node_Post_Processing.ipynb)"]:::postOrange
        ReRanking["Re-Ranking\n(06_02_ReRanking.ipynb)"]:::postOrange
        FLARE["FLARE\n(06_03_FLARE.ipynb)"]:::postOrange
        PromptComp["Prompt Compression\n(06_04_Prompt_Compression.ipynb)"]:::postOrange
        SelfCorrect["Self-Correcting\n(06_05_Self_Correcting.ipynb)"]:::postOrange
    end

    %% Modular RAG Extensions
    subgraph "Modular RAG Extensions"
        direction TB
        Hybrid["Hybrid Retrievers\n(07_01_Hybrid_Retrievers.ipynb)"]:::postOrange
        Agentic["Agentic RAG\n(07_02_Agentic_RAG.ipynb)"]:::postOrange
        EnsembleRet["Ensemble Retrieval\n(07_03_Ensemble_Retrieval.ipynb)"]:::postOrange
        EnsembleQuery["Ensemble Query Engine\n(07_04_Ensemble_Query_Engine.ipynb)"]:::postOrange
    end

    %% LLM & Evaluation
    LLM["LLM API\n(external)"]:::external
    subgraph "Evaluation"
        direction TB
        EvalIntro["RAG Evaluation Intro\n(04_01_Introduction_to_RAG_Evaluation.md)"]:::evalPurple
        Metrics["Evaluation Metrics\n(04_02_Evaluation_Metrics.md)"]:::evalPurple
        EvalSet["Create Eval Set\n(04_03_How_to_Create_an_Evaluation_Set.ipynb)"]:::evalPurple
    end

    %% Orchestration & Environment
    subgraph "Orchestration & Environment"
        direction TB
        DevContainer["Dev Container\n(.devcontainer/devcontainer.json)"]:::external
        CI["CI Workflow\n(.github/workflows/main.yml)"]:::external
        Requirements["Dependencies\n(requirements.txt)"]:::external
    end

    %% Connections
    DataSources --> TextCleaning
    TextCleaning --> Loading
    Utils --> Loading
    Loading --> Indexing
    Indexing --> Storing
    Storing --> IngestionOverview
    IngestionOverview --> VectorDB

    UserQuery --> QueryFund
    QueryFund --> QueryIntro
    QueryIntro --> PromptEng
    PromptEng --> RetrievedDocs["Retrieved Documents"]:::retrievalGreen
    RetrievedDocs --> NodeProc
    NodeProc --> ReRanking
    ReRanking --> FLARE
    FLARE --> PromptComp
    PromptComp --> SelfCorrect
    SelfCorrect --> FinalPrompt["Final Prompt"]:::retrievalGreen
    FinalPrompt --> LLM
    LLM --> ModelResponse["Model Response"]:::external
    ModelResponse --> EvalIntro
    EvalIntro --> Metrics
    Metrics --> EvalSet

    %% Orchestration arrows
    DevContainer -.-> DataSources
    DevContainer -.-> UserQuery
    DevContainer -.-> ModelResponse
    CI -.-> Loading
    CI -.-> QueryFund
    Requirements -.-> Indexing
    Requirements -.-> PromptEng

    %% Click Events
    click DataSources "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/01_Getting_Started/01_07_Downloading_Our_Data.ipynb"
    click TextCleaning "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/helpers/text_cleaning_helpers.py"
    click Utils "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/helpers/utils.py"
    click Loading "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/02_Fundamental_Concepts_in_LlamaIndex/02_03_Loading_Data.ipynb"
    click Indexing "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/02_Fundamental_Concepts_in_LlamaIndex/02_04_Indexing.ipynb"
    click Storing "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/02_Fundamental_Concepts_in_LlamaIndex/02_05_Storing.ipynb"
    click IngestionOverview "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/03_Introduction_to_RAG/03_02_Ingestion_Pipeline.ipynb"
    click VectorDB "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/.env_example"
    click QueryFund "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/02_Fundamental_Concepts_in_LlamaIndex/02_06_Querying.ipynb"
    click QueryIntro "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/03_Introduction_to_RAG/03_03_Query_Pipeline.ipynb"
    click PromptEng "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/03_Introduction_to_RAG/03_04_Prompt_Engineering_for_RAG.ipynb"
    click NodeProc "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/06_Advanced_RAG_Post_Retrieval/06_01_Node_Post_Processing.ipynb"
    click ReRanking "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/06_Advanced_RAG_Post_Retrieval/06_02_ReRanking.ipynb"
    click FLARE "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/06_Advanced_RAG_Post_Retrieval/06_03_FLARE.ipynb"
    click PromptComp "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/06_Advanced_RAG_Post_Retrieval/06_04_Prompt_Compression.ipynb"
    click SelfCorrect "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/06_Advanced_RAG_Post_Retrieval/06_05_Self_Correcting.ipynb"
    click Hybrid "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/07_Modular_RAG/07_01_Hybrid_Retrievers.ipynb"
    click Agentic "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/07_Modular_RAG/07_02_Agentic_RAG.ipynb"
    click EnsembleRet "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/07_Modular_RAG/07_03_Ensemble_Retrieval.ipynb"
    click EnsembleQuery "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/07_Modular_RAG/07_04_Ensemble_Query_Engine.ipynb"
    click EvalIntro "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/04_RAG_Evaluation/04_01_Introduction_to_RAG_Evaluation.md"
    click Metrics "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/04_RAG_Evaluation/04_02_Evaluation_Metrics.md"
    click EvalSet "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/04_RAG_Evaluation/04_03_How_to_Create_an_Evaluation_Set.ipynb"
    click DevContainer "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/.devcontainer/devcontainer.json"
    click CI "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/.github/workflows/main.yml"
    click Requirements "https://github.com/varunnthakuur/hands-on-ai-rag-using-llamaindex-3830207/blob/main/requirements.txt"

    %% Styles
    classDef dataBlue fill:#D0E8FF,stroke:#009,stroke-width:1px
    classDef ingestionBlue fill:#B3D7FF,stroke:#006,stroke-width:1px
    classDef retrievalGreen fill:#D2FFD0,stroke:#080,stroke-width:1px
    classDef postOrange fill:#FFD8B3,stroke:#E65,stroke-width:1px
    classDef evalPurple fill:#E8D0FF,stroke:#808,stroke-width:1px
    classDef storage fill:#C8FFE0,stroke:#080,stroke-width:1px
    classDef external fill:#FFE0E0,stroke:#800,stroke-width:1px
```
