# john-carroll-sw/coffee-chat-voice-assistant architecture

> Architecture diagram of the GitHub repository john-carroll-sw/coffee-chat-voice-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/john-carroll-sw/coffee-chat-voice-assistant
- Repository: https://github.com/john-carroll-sw/coffee-chat-voice-assistant

## Overview

##### Project Type and Purpose
- This is a full-stack voice-driven ordering web application.
- Purpose: capture live audio, send it to an Azure-powered backend for RAG (Retrieval-Augmented Generation) and real-time transcription, then render conversation and order state in the browser with live updates and speech synthesis.

##### File Structure
- Top-level “app” directory splits into:
- frontend (React + TypeScript + Vite + TailwindCSS)
- backend (Python, Flask/FastAPI-style app.py plus Azure Speech, OpenAI RAG tools)
- “infra” directory holds Bicep templates to provision Azure Container Apps, identity, search, registry, roles.
- “scripts” directory contains local start, build, deploy tooling.

##### README for Insights
- Confirms key components: RTClient (frontend audio sender), RTMiddleTier (backend Azure OpenAI realtime API wrapper), Azure AI Search tool, on-the-fly function calling, real-time transcription & speech synthesis.
- References Dockerfile to package both frontend and backend into a single container for Azure Container Apps.

##### Main Components (to represent
a. User’s Browser
- Audio capture (Web Audio API, audio-processor-worklet.js)
- React App / RTClient
- Live UI: transcript panel, order summary, status messages
b. Backend Container (Python service)
- RTMiddleTier: streams audio to Azure OpenAI GPT-4o Realtime API
- AzureSpeech module: optional speech-to-text or text-to-speech
- RAG tool: queries Azure AI Search index for menu grounding
- Order-state & Models: manages conversation state and function calls
c. Azure Services
- Azure OpenAI (GPT-4o real-time endpoint)
- Azure Cognitive Search (index of menu items ingested via notebooks)
- Azure Container Registry (stores Docker image)
- Azure Container Apps (hosts backend + static frontend)
- Azure Key Vault (for secrets, if in scope)
d. CI/CD / Infra
- GitHub Actions workflows (azure-dev.yaml)
- Bicep templates under infra/core for infra provisioning

##### Relationships and Data Flows
1. Browser audio → RTClient → Backend (HTTP/WebSocket)
2. Backend → Azure OpenAI (streaming audio/text)
3. Backend → Azure AI Search (JSON search queries)
4. Azure AI Search → Backend (search results)
5. OpenAI responses → Backend function-call resolution → Browser (UI update + TTS)
6. Backend synthesizes audio via AzureSpeech → Browser audio playback
- Flow for menu ingestion notebooks → Azure AI Search index.

##### Architectural Patterns & Design Principles
- Real-Time RAG pattern: frontend RTClient ↔ RTMiddleTier ↔ Azure OpenAI + Azure Search
- Separation of concerns: UI components vs. streaming logic vs. RAG vs. state management
- Containerized microservices: one container hosts both frontend and backend
- Infrastructure as Code: Bicep modules for repeatable Azure deployment
- Extensibility: Jupyter notebooks for data ingestion.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Browser (Frontend)"
        direction TB
        AWP["Audio Processor Worklet"]:::frontend
        AWP2["Audio Playback Worklet"]:::frontend
        RC["React RTClient App"]:::frontend
        REC["Recorder Component"]:::frontend
        PL["Player Component"]:::frontend
        TP["Transcript Panel"]:::frontend
        OS["Order Summary Panel"]:::frontend
        SM["Status Message Panel"]:::frontend
        UA["useAudioRecorder Hook"]:::frontend
        UP["useAudioPlayer Hook"]:::frontend
        ASH["useAzureSpeech Hook"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Service (Containerized)" 
        direction TB
        AP["app.py (Service Entry)"]:::backend
        RTM["RTMiddleTier API Wrapper"]:::backend
        AZS["azurespeech.py"]:::backend
        GPTM["azure_speech_gpt4o_mini.py"]:::backend
        RAG["tools.py (RAG Tool)"]:::backend
        OSM["order_state.py"]:::backend
        MOD["models.py"]:::backend
    end

    %% Azure Services
    subgraph "Azure Services"
        direction TB
        OAI["Azure OpenAI (GPT-4o)"]:::azure
        ACS["Azure Cognitive Search"]:::azure
        ACR["Azure Container Registry"]:::azure
        ACA["Azure Container Apps"]:::azure
        AKV["Azure Key Vault"]:::azure
    end

    %% Menu Ingestion
    subgraph "Data Ingestion"
        direction TB
        NJ["menus ingestion JSON Notebook"]:::infra
        PJ["menus ingestion PDF Notebook"]:::infra
    end

    %% CI/CD & Infra
    subgraph "CI/CD & Infra"
        direction TB
        GH["GitHub Actions Workflow"]:::infra
        BP["Bicep Templates"]:::infra
    end

    %% Runtime Data Flows
    AWP -->|"audio frames"| RC
    AWP2 -->|"playback sink"| PL
    RC -->|"WebSocket/HTTP"| AP
    AP -->|"stream audio/text"| RTM
    RTM -->|"Azure SDK call"| OAI
    AP -->|"search query JSON"| ACS
    ACS -->|"search results"| AP
    OAI -->|"stream responses"| RTM
    RTM -->|"function calls & order logic"| AP
    AP -->|"UI update + TTS text"| RC
    AP -->|"synthesize audio"| AZS
    AZS -->|"audio stream"| RC

    %% Menu Ingestion Flow
    NJ -->|"ingest data"| ACS
    PJ -->|"ingest data"| ACS

    %% Infra Deployment Flow (dashed)
    GH -.->|"deploy infra"| BP
    BP -.->|"provision"| ACR
    BP -.->|"provision"| ACA
    BP -.->|"provision"| ACS
    BP -.->|"provision"| AKV

    %% Click Events Frontend
    click AWP "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/public/audio-processor-worklet.js"
    click AWP2 "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/public/audio-playback-worklet.js"
    click RC "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/App.tsx"
    click RC "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/hooks/useRealtime.tsx"
    click REC "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/components/audio/recorder.ts"
    click PL "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/components/audio/player.ts"
    click TP "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/components/ui/transcript-panel.tsx"
    click OS "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/components/ui/order-summary.tsx"
    click SM "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/components/ui/status-message.tsx"
    click UA "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/hooks/useAudioRecorder.tsx"
    click UP "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/hooks/useAudioPlayer.tsx"
    click ASH "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/frontend/src/hooks/useAzureSpeech.tsx"

    %% Click Events Backend
    click AP "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/app.py"
    click RTM "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/rtmt.py"
    click AZS "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/azurespeech.py"
    click GPTM "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/azure_speech_gpt4o_mini.py"
    click RAG "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/tools.py"
    click OSM "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/order_state.py"
    click MOD "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/app/backend/models.py"
    click ACR "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/infra/core/host/container-app.bicep"
    click BP "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/infra/main.bicep"
    click GH "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/.github/workflows/azure-dev.yaml"
    click NJ "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/scripts/menu_ingestion_search_json.ipynb"
    click PJ "https://github.com/john-carroll-sw/coffee-chat-voice-assistant/blob/main/scripts/menu_ingestion_search_pdf.ipynb"

    %% Styles
    classDef frontend fill:#ccf,stroke:#3399ff,color:#000;
    classDef backend fill:#cfc,stroke:#33aa33,color:#000;
    classDef azure fill:#fca,stroke:#ff9933,color:#000;
    classDef infra fill:#eee,stroke:#999,color:#000;
```
