How azure/gpt-rag-ingestion works
Main Components
• Ingestion Trigger & Azure Functions: At the top level, the Function App (“function_app.py”, “host.json”) contains several functions that trigger different processes:
• Document ingestion from blob storage (“ragindex-indexer-chunk-documents”).
• SharePoint file indexing and purging (run from “run_sharepoint.py” and within the connectors/sharepoint directory).
• Document Chunking Module: The “chunking” directory contains the core business logic for splitting documents. It encapsulates multiple chunkers for different file types (e.g., DocAnalysisChunker, LangChainChunker, TranscriptionChunker, SpreadsheetChunker).
• Connectors: Under “connectors”, components are designed to integrate with external sources such as SharePoint (and also an image deleted files purger).
• External Services & Dependencies: Represent external Azure resources including:
• Blob Storage (for document files and images),
• Azure AI Search (for indexing document chunks),
• Azure OpenAI (for generating embeddings),
• Document Intelligence (for PDF, DOCX, PPTX analysis and OCR),
• Microsoft Graph API (for SharePoint integration),
• Azure Key Vault (for secrets management).
Relationships and Interactions
• The data ingestion pipeline begins by reading documents from a designated blob container (“documents”).
• Each document is then processed by the appropriate chunker in the “chunking” module:
• File extension routes the processing to a specific chunker (e.g., .pdf uses DocAnalysisChunker, .vtt uses TranscriptionChunker, .xlsx uses SpreadsheetChunker, etc.).
• In the multimodal ingestion path, if the “MULTIMODAL” environment variable is flagged true, the pipeline utilizes the MultimodalChunker to extract images, perform OCR, caption images, and combine these captions with text.
• After chunking, the Azure Functions call the embedding generation service using the configured Azure OpenAI models.
• The output chunks (with embedded vectors) are then stored/indexed into the Azure AI Search Index.
• For SharePoint integration, a dedicated connector fetches files via the Microsoft Graph API, processes them (using the same chunking pipeline), and updates the search index. Separately, a purging connector cleans up outdated or deleted items in the index.
Architectural Patterns and Design Principles
• Function-as-a-Service (FaaS) – The ingestion component is built on Azure Functions ensuring serverless, event-driven processing.
• Modular Design – The chunking functionality is abstracted into interchangeable components (each file type has its own dedicated chunker in the “chunking/chunkers” directory) which allows for easy extension or customization.
• Connector Abstraction – External system interactions (e.g., SharePoint) are encapsulated in the “connectors” folder.
• Configuration & Environment-Driven Behavior – Several settings (found in app_settings.json, .env.template, and local.settings.json.template) direct the runtime behavior, service endpoints, and integration details.
• Microservices or Pipeline Pattern – The overall flow is decoupled: ingestion, chunking, embedding generation, and indexing represent discrete, interconnected stages.
Specify Technologies, Frameworks, and Libraries
• Programming Language – Python (the Function App and all components are in Python).
• Azure Services – Functions, Blob Storage, AI Search, OpenAI, Document Intelligence, and Key Vault.
• External Libraries – LangChain (for text chunking), Microsoft Graph API for SharePoint connectivity, and additional utility modules (in “utils” and “tools”).