How john-carroll-sw/coffee-chat-voice-assistant works
Project Type and Purpose
• This is a full-stack voice-driven ordering web application.
• Purpose: capture live audio, send it to an Azure-powered backend for RAG (Retrieval-Augmented Generation) and real-time transcription, then render conversation and order state in the browser with live updates and speech synthesis.
File Structure
• Top-level “app” directory splits into:
• frontend (React + TypeScript + Vite + TailwindCSS)
• backend (Python, Flask/FastAPI-style app.py plus Azure Speech, OpenAI RAG tools)
• “infra” directory holds Bicep templates to provision Azure Container Apps, identity, search, registry, roles.
• “scripts” directory contains local start, build, deploy tooling.
README for Insights
• Confirms key components: RTClient (frontend audio sender), RTMiddleTier (backend Azure OpenAI realtime API wrapper), Azure AI Search tool, on-the-fly function calling, real-time transcription & speech synthesis.
• References Dockerfile to package both frontend and backend into a single container for Azure Container Apps.
Main Components (to represent
a. User’s Browser
• Audio capture (Web Audio API, audio-processor-worklet.js)
• React App / RTClient
• Live UI: transcript panel, order summary, status messages
b. Backend Container (Python service)
• RTMiddleTier: streams audio to Azure OpenAI GPT-4o Realtime API
• AzureSpeech module: optional speech-to-text or text-to-speech
• RAG tool: queries Azure AI Search index for menu grounding
• Order-state & Models: manages conversation state and function calls
c. Azure Services
• Azure OpenAI (GPT-4o real-time endpoint)
• Azure Cognitive Search (index of menu items ingested via notebooks)
• Azure Container Registry (stores Docker image)
• Azure Container Apps (hosts backend + static frontend)
• Azure Key Vault (for secrets, if in scope)
d. CI/CD / Infra
• GitHub Actions workflows (azure-dev.yaml)
• Bicep templates under infra/core for infra provisioning
Relationships and Data Flows
1. Browser audio → RTClient → Backend (HTTP/WebSocket)
2. Backend → Azure OpenAI (streaming audio/text)
3. Backend → Azure AI Search (JSON search queries)
4. Azure AI Search → Backend (search results)
5. OpenAI responses → Backend function-call resolution → Browser (UI update + TTS)
6. Backend synthesizes audio via AzureSpeech → Browser audio playback
• Flow for menu ingestion notebooks → Azure AI Search index.
Architectural Patterns & Design Principles
• Real-Time RAG pattern: frontend RTClient ↔ RTMiddleTier ↔ Azure OpenAI + Azure Search
• Separation of concerns: UI components vs. streaming logic vs. RAG vs. state management
• Containerized microservices: one container hosts both frontend and backend
• Infrastructure as Code: Bicep modules for repeatable Azure deployment
• Extensibility: Jupyter notebooks for data ingestion.