# azure/gpt-rag-ingestion architecture

> Architecture diagram of the GitHub repository azure/gpt-rag-ingestion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure/gpt-rag-ingestion
- Repository: https://github.com/azure/gpt-rag-ingestion

## Overview

##### Main Components
- Ingestion Trigger & Azure Functions: At the top level, the Function App (“function_app.py”, “host.json”) contains several functions that trigger different processes:
- Document ingestion from blob storage (“ragindex-indexer-chunk-documents”).
- SharePoint file indexing and purging (run from “run_sharepoint.py” and within the connectors/sharepoint directory).
- Document Chunking Module: The “chunking” directory contains the core business logic for splitting documents. It encapsulates multiple chunkers for different file types (e.g., DocAnalysisChunker, LangChainChunker, TranscriptionChunker, SpreadsheetChunker).
- Connectors: Under “connectors”, components are designed to integrate with external sources such as SharePoint (and also an image deleted files purger).
- External Services & Dependencies: Represent external Azure resources including:
- Blob Storage (for document files and images),
- Azure AI Search (for indexing document chunks),
- Azure OpenAI (for generating embeddings),
- Document Intelligence (for PDF, DOCX, PPTX analysis and OCR),
- Microsoft Graph API (for SharePoint integration),
- Azure Key Vault (for secrets management).

##### Relationships and Interactions
- The data ingestion pipeline begins by reading documents from a designated blob container (“documents”).
- Each document is then processed by the appropriate chunker in the “chunking” module:
- File extension routes the processing to a specific chunker (e.g., .pdf uses DocAnalysisChunker, .vtt uses TranscriptionChunker, .xlsx uses SpreadsheetChunker, etc.).
- In the multimodal ingestion path, if the “MULTIMODAL” environment variable is flagged true, the pipeline utilizes the MultimodalChunker to extract images, perform OCR, caption images, and combine these captions with text.
- After chunking, the Azure Functions call the embedding generation service using the configured Azure OpenAI models.
- The output chunks (with embedded vectors) are then stored/indexed into the Azure AI Search Index.
- For SharePoint integration, a dedicated connector fetches files via the Microsoft Graph API, processes them (using the same chunking pipeline), and updates the search index. Separately, a purging connector cleans up outdated or deleted items in the index.

##### Architectural Patterns and Design Principles
- Function-as-a-Service (FaaS) – The ingestion component is built on Azure Functions ensuring serverless, event-driven processing.
- Modular Design – The chunking functionality is abstracted into interchangeable components (each file type has its own dedicated chunker in the “chunking/chunkers” directory) which allows for easy extension or customization.
- Connector Abstraction – External system interactions (e.g., SharePoint) are encapsulated in the “connectors” folder.
- Configuration & Environment-Driven Behavior – Several settings (found in app_settings.json, .env.template, and local.settings.json.template) direct the runtime behavior, service endpoints, and integration details.
- Microservices or Pipeline Pattern – The overall flow is decoupled: ingestion, chunking, embedding generation, and indexing represent discrete, interconnected stages.

##### Specify Technologies, Frameworks, and Libraries
- Programming Language – Python (the Function App and all components are in Python).
- Azure Services – Functions, Blob Storage, AI Search, OpenAI, Document Intelligence, and Key Vault.
- External Libraries – LangChain (for text chunking), Microsoft Graph API for SharePoint connectivity, and additional utility modules (in “utils” and “tools”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    Blob_Storage["Azure Blob Storage"]:::data
    SP_External["SharePoint (MSGraph)"]:::external

    %% Ingestion Functions Subgraph
    subgraph "Azure Functions"
        Function_App["Function App"]:::function
        Host_Config["Host Config"]:::function
    end

    %% Document Chunking Module Subgraph
    subgraph "Document Chunking Module"
        Document_Chunking_Core["Document Chunking Core"]:::module
        Chunkers["Chunkers"]:::module
    end

    %% Processing & Indexing
    Embedding_Generation["Embedding Generation (Azure OpenAI)"]:::externalService
    AI_Search["Azure AI Search Index"]:::data

    %% Connectors Subgraph
    subgraph "SharePoint Connector"
        SP_Files_Indexer["SP Files Indexer"]:::connector
        SP_Data_Reader["SP Data Reader"]:::connector
        SP_Deleted_Files_Purger["SP Deleted Files Purger"]:::connector
        Run_SharePoint["SharePoint Connector (run_sharepoint.py)"]:::connector
    end
    Image_Deleted_Files_Purger["Image Deleted Files Purger"]:::connector

    %% Configuration & Utilities Subgraph
    subgraph "Configuration & Environment"
        App_Settings["app_settings.json"]:::config
        Local_Settings["local.settings.json.template"]:::config
        Env_Template[".env.template"]:::config
    end

    subgraph "External Service Utilities"
        AOAI["AOAI"]:::tools
        Doc_Intelligence["Doc Intelligence"]:::tools
        KeyVault["KeyVault"]:::tools
        File_Utils["File Utils"]:::tools
    end

    %% Data Flows
    Blob_Storage -->|"trigger"| Function_App
    Function_App -->|"calls"| Document_Chunking_Core
    Host_Config ---|"configures"| Function_App

    Document_Chunking_Core -->|"processes"| Chunkers
    Chunkers -->|"generates chunks"| Embedding_Generation
    Embedding_Generation -->|"indexes documents"| AI_Search

    %% SharePoint Connector Flows
    SP_External -->|"fetch files"| SP_Files_Indexer
    SP_External -->|"fetch data"| SP_Data_Reader
    SP_Files_Indexer -->|"routes to"| Run_SharePoint
    SP_Data_Reader -->|"routes to"| Run_SharePoint
    Run_SharePoint -->|"uses pipeline"| Document_Chunking_Core
    SP_Deleted_Files_Purger ---|"purges index"| AI_Search

    %% Image Purger Flow
    Image_Deleted_Files_Purger ---|"cleans images"| AI_Search

    %% External Dependencies
    App_Settings ---|"configures"| Function_App
    Local_Settings ---|"configures"| Function_App
    Env_Template ---|"configures"| Function_App
    AOAI ---|"supports"| Embedding_Generation
    Doc_Intelligence ---|"supports"| Embedding_Generation
    KeyVault ---|"secures"| Function_App
    File_Utils ---|"assists"| Function_App

    %% Click Events for Ingestion Functions
    click Function_App "https://github.com/azure/gpt-rag-ingestion/blob/main/function_app.py"
    click Host_Config "https://github.com/azure/gpt-rag-ingestion/blob/main/host.json"

    %% Click Events for Document Chunking Module
    click Document_Chunking_Core "https://github.com/azure/gpt-rag-ingestion/blob/main/chunking/document_chunking.py"
    click Chunkers "https://github.com/azure/gpt-rag-ingestion/tree/main/chunking/chunkers/"

    %% Click Events for SharePoint Connector
    click SP_Files_Indexer "https://github.com/azure/gpt-rag-ingestion/blob/main/connectors/sharepoint/sharepoint_files_indexer.py"
    click SP_Data_Reader "https://github.com/azure/gpt-rag-ingestion/blob/main/connectors/sharepoint/sharepoint_data_reader.py"
    click SP_Deleted_Files_Purger "https://github.com/azure/gpt-rag-ingestion/blob/main/connectors/sharepoint/sharepoint_deleted_files_purger.py"
    click Run_SharePoint "https://github.com/azure/gpt-rag-ingestion/blob/main/run_sharepoint.py"

    %% Click Event for Image Purger
    click Image_Deleted_Files_Purger "https://github.com/azure/gpt-rag-ingestion/blob/main/connectors/images_deleted_files_purger.py"

    %% Click Events for Configuration & Environment
    click App_Settings "https://github.com/azure/gpt-rag-ingestion/blob/main/app_settings.json"
    click Local_Settings "https://github.com/azure/gpt-rag-ingestion/blob/main/local.settings.json.template"
    click Env_Template "https://github.com/azure/gpt-rag-ingestion/blob/main/.env.template"

    %% Click Events for External Service Utilities
    click AOAI "https://github.com/azure/gpt-rag-ingestion/blob/main/tools/aoai.py"
    click Doc_Intelligence "https://github.com/azure/gpt-rag-ingestion/blob/main/tools/doc_intelligence.py"
    click KeyVault "https://github.com/azure/gpt-rag-ingestion/blob/main/tools/keyvault.py"
    click File_Utils "https://github.com/azure/gpt-rag-ingestion/blob/main/utils/file_utils.py"

    %% Styles
    classDef function fill:#F4D03F,stroke:#333,stroke-width:2px;
    classDef module fill:#85C1E9,stroke:#333,stroke-width:2px;
    classDef connector fill:#BB8FCE,stroke:#333,stroke-width:2px;
    classDef data fill:#82E0AA,stroke:#333,stroke-width:2px;
    classDef external fill:#E74C3C,stroke:#333,stroke-width:2px;
    classDef externalService fill:#F1948A,stroke:#333,stroke-width:2px;
    classDef config fill:#D5F5E3,stroke:#333,stroke-width:2px;
    classDef tools fill:#FAD7A0,stroke:#333,stroke-width:2px;
```
