How azure-samples/aisearch-openai-rag-audio works
Project type and purpose
• This is a full-stack, voice-enabled Retrieval-Augmented Generation (RAG) application.
• The frontend is a React/TypeScript web app that captures and plays back audio.
• The backend is a Python service that orchestrates audio streaming, Azure AI Search calls, and real-time calls to the GPT-4o audio API.
• The infra folder supplies Azure Bicep templates to provision Container Apps, Search, OpenAI, identities, and registry.
Core directories and components
• app/frontend: React + Vite (+ Tailwind) + Web Audio Worklet for microphone capture (useAudioRecorder) and playback (useAudioPlayer).
• app/backend: Python Flask (or FastAPI‐style) app (app.py) with modules:
• rtmt.py / ragtools.py / setup_intvect.py – glue for real-time API, RAG search, and vector ingestion.
• requirements.txt and .env for Azure credentials and endpoints.
• infra: Bicep scripts to deploy Azure Container Apps (host), Azure AI Search indices, managed identities, and role assignments.
• scripts: Shell/PowerShell scripts to set up local env and launch the app.
Key external services
• Azure OpenAI GPT-4o Realtime API (via WebSocket).
• Azure AI Search (for RAG).
• (Optionally) Azure Blob Storage to host PDF documents before indexing.
• Azure Container Registry and Container Apps (for hosting frontend & backend).
Architectural patterns and design principles
• Separation of concerns: Frontend handles UI and audio I/O; backend handles business logic, streaming, and service orchestration.
• Streaming pattern: Bi-directional WebSocket between frontend RTClient and backend RTMiddleTier for chunked audio in/out.
• Tooling pattern for LLM: Backend uses a “Search” tool (Azure Search) to retrieve docs, then passes context + audio stream to OpenAI.
• Infrastructure as code: Declarative Bicep templates for reproducible Azure deployment.
Diagramming conventions
• Green = real-time audio WebSocket traffic
• Blue = search/HTTP REST calls
• Orange = IaC/Deployment
• Clouds for managed Azure services
• Rectangles for compute (frontend, backend)