How azure-samples/azure-open-ai-rag-oyd-text-images works
Project Type and Purpose
• This is a full-stack, serverless RAG (Retrieval-Augmented Generation) application specialized for PDF documents containing both text and embedded images.
• Purpose: Ingest PDFs, extract text chunks and images, index them via Azure AI Search with embeddings, and serve a chat-style demo (web app or notebook) that returns LLM responses with text and image citations.
Top-Level Components (based on file tree & README)
a. Infrastructure as Code
• Bicep files (bicep/, bicep/helpers/, helper.sh) define Azure resources: Blob Storage, Function App, Cognitive Search, OpenAI deployments.
b. Data Ingestion Pipeline
• Azure Blob Storage: raw_data/ and prepared_data/ paths.
• Azure Function (azure-function/): Python code triggered on blob upload to split PDFs, extract text & images, and write processed artifacts back to blob.
c. Indexing Service
• Azure AI Search (azure-ai-search referenced in README): configured with data source, skillset, index, and indexer to vectorize text via OAI embedding model and link images to text chunks.
d. Application Runtime
• Demo Web App (demo-app/): Python (Streamlit) front end and server-side chat logic that calls Azure OpenAI (chat + embedding deployments) and retrieves context from AI Search.
• Demo Notebook (demo-notebook/): Jupyter notebook version mirroring the client/server interactions.
e. Supporting Artifacts
• CI/CD workflows (.github/workflows/hugo.yml for docs/site), Dockerfile for containerizing demo environment, workshop content (workshop/).
Relationships and Data Flows
• PDF Upload → Blob Storage (raw_data)
• Blob Trigger → Azure Function
• Extract text chunks + images → upload to Blob Storage (prepared_data)
• Scheduled/On-demand → AI Search Indexer
• Reads from prepared_data → embeds text via OAI → populates Search Index with text + image metadata
• User Query → Demo App Client
• Forwards to Server Chat logic → consults Azure OpenAI (chat) which under the hood queries AI Search for context → returns enriched response with citations
• Demo App Server → Blob Storage
• Serves image URLs for cited images back to client UI
• Client UI → Renders text + image citations
Architectural Patterns & Principles
• Event-driven pipeline: Blob storage events triggering Functions
• Serverless compute: Azure Functions + managed Azure AI Search + OpenAI service
• Separation of concerns: ingestion, indexing, AI chat, UI layers
• Infrastructure as Code: declarative Bicep definitions, reproducible deployments
• Extensibility: workshop content and helper scripts for learning and customization
Final Advice
• Don’t omit the workshop and notebook; indicate them as educational overlays.
• Extension points: where users could plug in custom PDFs or swap embedding models.