# azure-samples/aisearch-openai-rag-audio architecture

> Architecture diagram of the GitHub repository azure-samples/aisearch-openai-rag-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure-samples/aisearch-openai-rag-audio
- Repository: https://github.com/azure-samples/aisearch-openai-rag-audio

## Overview

##### Project type and purpose
- This is a full-stack, voice-enabled Retrieval-Augmented Generation (RAG) application.
- The frontend is a React/TypeScript web app that captures and plays back audio.
- The backend is a Python service that orchestrates audio streaming, Azure AI Search calls, and real-time calls to the GPT-4o audio API.
- The infra folder supplies Azure Bicep templates to provision Container Apps, Search, OpenAI, identities, and registry.

##### Core directories and components
- **app/frontend**: React + Vite (+ Tailwind) + Web Audio Worklet for microphone capture (`useAudioRecorder`) and playback (`useAudioPlayer`).
- **app/backend**: Python Flask (or FastAPI‐style) app (`app.py`) with modules:
- `rtmt.py` / `ragtools.py` / `setup_intvect.py` – glue for real-time API, RAG search, and vector ingestion.
- `requirements.txt` and `.env` for Azure credentials and endpoints.
- **infra**: Bicep scripts to deploy Azure Container Apps (host), Azure AI Search indices, managed identities, and role assignments.
- **scripts**: Shell/PowerShell scripts to set up local env and launch the app.

##### Key external services
- Azure OpenAI GPT-4o Realtime API (via WebSocket).
- Azure AI Search (for RAG).
- (Optionally) Azure Blob Storage to host PDF documents before indexing.
- Azure Container Registry and Container Apps (for hosting frontend & backend).

##### Architectural patterns and design principles
- **Separation of concerns**: Frontend handles UI and audio I/O; backend handles business logic, streaming, and service orchestration.
- **Streaming pattern**: Bi-directional WebSocket between frontend RTClient and backend RTMiddleTier for chunked audio in/out.
- **Tooling pattern for LLM**: Backend uses a “Search” tool (Azure Search) to retrieve docs, then passes context + audio stream to OpenAI.
- **Infrastructure as code**: Declarative Bicep templates for reproducible Azure deployment.

##### Diagramming conventions
- Green = real-time audio WebSocket traffic
- Blue = search/HTTP REST calls
- Orange = IaC/Deployment
- Clouds for managed Azure services
- Rectangles for compute (frontend, backend)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and Frontend Layer
    User["User Browser\n(Microphone & Audio Player)"]:::frontend
    subgraph "Frontend Layer"
        direction TB
        FEApp["React App\n(React/TypeScript/Vite/Tailwind)"]:::frontend
        Recorder["useAudioRecorder Hook"]:::frontend
        Player["useAudioPlayer Hook"]:::frontend
        RTClient["useRealtime Hook\n(WebSocket Client)"]:::frontend
        WorkletCap["audio-processor-worklet.js"]:::frontend
        WorkletPlay["audio-playback-worklet.js"]:::frontend
        UIComp["UI Components\n(Buttons, Panels, Cards)"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        direction TB
        PyApp["Python Backend\n(app.py)"]:::backend
        RTMT["RTMiddleTier\n(rtmt.py)"]:::backend
        RAG["RAG Tooling\n(ragtools.py)"]:::backend
        SetupVec["Vector Ingestion\n(setup_intvect.py)"]:::backend
        EnvDeps["requirements.txt & .env"]:::backend
    end

    %% External Data Stores and Services
    subgraph "Azure Services"
        direction TB
        SearchCloud["Azure AI Search"]:::azure
        SearchIndex["Search Index"]:::datastore
        BlobStore["Blob Storage"]:::datastore
        OpenAI["Azure OpenAI GPT-4o Audio\n(Realtime API)"]:::azure
    end

    %% Infrastructure Layer
    subgraph "Infrastructure IaC"
        direction TB
        InfraDir["infra"]:::infra
        HostMod["Host Module\n(core/host)"]:::infra
        SecMod["Security Module\n(core/security)"]:::infra
        MainBicep["main.bicep"]:::infra
        Params["main.parameters.json"]:::infra
        Scripts["scripts"]:::infra
    end

    %% Data Flows
    User -->|speaks to| Recorder
    Recorder -->|audio frames| RTClient
    RTClient -->|WebSocket audio| RTMT
    RTMT -->|REST Search| SearchCloud
    SearchCloud -->|returns docs| SearchIndex
    SearchIndex -->|docs| RTMT
    RTMT -->|WebSocket audio + context| OpenAI
    OpenAI -->|audio chunks + text| RTMT
    RTMT -->|WebSocket audio| RTClient
    RTClient -->|audio frames| Player
    RTMT -->|JSON transcripts & citations| FEApp
    FEApp -->|render UI| UIComp
    FEApp -->|loads hooks & worklets| Recorder
    FEApp -->|loads hooks & worklets| Player
    FEApp -->|loads hooks & worklets| RTClient
    FEApp -->|loads worklets| WorkletCap
    FEApp -->|loads worklets| WorkletPlay

    %% IaC Deployments (dashed)
    InfraDir -.->|deploy| HostMod
    InfraDir -.->|deploy| SecMod
    InfraDir -.->|deploy main| MainBicep
    MainBicep -.->|params| Params
    InfraDir -.->|runs| Scripts

    %% Click Events
    click FEApp "https://github.com/azure-samples/aisearch-openai-rag-audio/tree/main/app/frontend"
    click Recorder "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/frontend/src/hooks/useAudioRecorder.tsx"
    click Player "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/frontend/src/hooks/useAudioPlayer.tsx"
    click RTClient "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/frontend/src/hooks/useRealtime.tsx"
    click WorkletCap "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/frontend/public/audio-processor-worklet.js"
    click WorkletPlay "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/frontend/public/audio-playback-worklet.js"
    click UIComp "https://github.com/azure-samples/aisearch-openai-rag-audio/tree/main/app/frontend/src/components/ui"
    click PyApp "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/backend/app.py"
    click RTMT "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/backend/rtmt.py"
    click RAG "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/backend/ragtools.py"
    click SetupVec "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/backend/setup_intvect.py"
    click EnvDeps "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/backend/requirements.txt"
    click EnvDeps "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/.env"
    click InfraDir "https://github.com/azure-samples/aisearch-openai-rag-audio/tree/main/infra"
    click HostMod "https://github.com/azure-samples/aisearch-openai-rag-audio/tree/main/infra/core/host"
    click SecMod "https://github.com/azure-samples/aisearch-openai-rag-audio/tree/main/infra/core/security"
    click MainBicep "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/infra/main.bicep"
    click Params "https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/infra/main.parameters.json"
    click Scripts "https://github.com/azure-samples/aisearch-openai-rag-audio/tree/main/scripts"

    %% Styles
    classDef frontend fill:#E8F4FD,stroke:#0275d8,color:#0275d8
    classDef backend fill:#F0F0F0,stroke:#5A5A5A,color:#5A5A5A
    classDef azure fill:#E0F7FA,stroke:#00ACC1,color:#006064
    classDef infra fill:#FFF3E0,stroke:#FB8C00,color:#E65100
    classDef datastore fill:#FFFDE7,stroke:#FBC02D,color:#F57F17
```
