How openai/openai-voice-agent-sdk-sample works
Project type and purpose
• This is a full‑stack sample application for building a conversational “voice agent.”
• Frontend: Next.js + React + Tailwind CSS, provides a push‑to‑talk UI, displays chat bubbles, loading indicators, function call messages, etc.
• Backend: Python FastAPI, hosts a WebSocket endpoint that manages streaming audio & text, integrates with the OpenAI Agents SDK, and optionally calls mock or real tools.
File structure
• Top‑level:
• frontend/ – all UI code, WebSocket client logic (useWebsocket.ts), audio hooks (useAudio.ts).
• server/ – FastAPI app (server/app), configuration (agent_config.py), mock API endpoints, utils, plus server.py to start Uvicorn.
• Makefile – orchestrates install and run (make sync, make serve).
• Patterns:
• Client-server separation via WebSocket.
• React components organized by feature (AudioChat, ChatDialog, MessageBubble, function‑call messages, etc.).
• FastAPI app routes and background agents SDK logic.
README insights
• Emphasizes WebSocket streaming, multi‑turn dialogue, function calling, tool invocation.
• No database layer—state is ephemeral or managed in memory via the Agents SDK.
a. Main components:
1. Browser/Client (Next.js app)
– UI layer: AudioChat component, ChatDialog, composer, playback.
– WebSocket client hook (useWebsocket) for sending/receiving messages and audio chunks.
– Audio capture & playback hook (useAudio).
2. FastAPI WebSocket Server
– WebSocket endpoint: receives audio/text events, routes them to agent logic.
– Agents SDK core: orchestrates calls to OpenAI API, handles streaming responses & tool calls.
– Tool adapters: mock_api or real REST calls, function‑calling stubs.
3. OpenAI API (external)
– Chat completion, function calling, speech‑to‑text/ text‑to‑speech if enabled.
4. (Optional) Mock API / External tools
– Sample endpoints in server/app/mock_api.py representing external services the agent can call.
b. Relationships and interactions:
• Client ↔ WebSocket Server: bi‑directional, streams JSON and binary audio frames.
• Server ↔ OpenAI API: HTTP(s) streaming for chat completions, function‑call invocations.
• Server ↔ Mock API: HTTP calls for tool execution.
• Client: renders server’s streamed responses, plays back TTS audio.
c. Architectural patterns & principles:
– Client-Server separation via persistent WebSocket for low‑latency streaming.
– Hooks in frontend to encapsulate cross‑cutting concerns (audio, websocket).
– Agents SDK on server to centralize conversational logic and tool orchestration.
– React component decomposition for UI modularity.
d. Key technologies/frameworks:
– Next.js, React, Tailwind CSS, WebSocket API on frontend.
– FastAPI, Uvicorn, Python Agents SDK on backend.
– OpenAI REST API as external dependency.