# openai/openai-voice-agent-sdk-sample architecture

> Architecture diagram of the GitHub repository openai/openai-voice-agent-sdk-sample, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/openai-voice-agent-sdk-sample
- Repository: https://github.com/openai/openai-voice-agent-sdk-sample

## Overview

##### Project type and purpose
- This is a full‑stack sample application for building a conversational “voice agent.”
- Frontend: Next.js + React + Tailwind CSS, provides a push‑to‑talk UI, displays chat bubbles, loading indicators, function call messages, etc.
- Backend: Python FastAPI, hosts a WebSocket endpoint that manages streaming audio & text, integrates with the OpenAI Agents SDK, and optionally calls mock or real tools.

##### File structure
- Top‑level:
- frontend/ – all UI code, WebSocket client logic (useWebsocket.ts), audio hooks (useAudio.ts).
- server/ – FastAPI app (server/app), configuration (agent_config.py), mock API endpoints, utils, plus server.py to start Uvicorn.
- Makefile – orchestrates install and run (`make sync`, `make serve`).
- Patterns:
- Client-server separation via WebSocket.
- React components organized by feature (AudioChat, ChatDialog, MessageBubble, function‑call messages, etc.).
- FastAPI app routes and background agents SDK logic.

##### README insights
- Emphasizes WebSocket streaming, multi‑turn dialogue, function calling, tool invocation.
- No database layer—state is ephemeral or managed in memory via the Agents SDK.

a. Main components:
1. Browser/Client (Next.js app)
– UI layer: AudioChat component, ChatDialog, composer, playback.
– WebSocket client hook (useWebsocket) for sending/receiving messages and audio chunks.
– Audio capture & playback hook (useAudio).
2. FastAPI WebSocket Server
– WebSocket endpoint: receives audio/text events, routes them to agent logic.
– Agents SDK core: orchestrates calls to OpenAI API, handles streaming responses & tool calls.
– Tool adapters: mock_api or real REST calls, function‑calling stubs.
3. OpenAI API (external)
– Chat completion, function calling, speech‑to‑text/ text‑to‑speech if enabled.
4. (Optional) Mock API / External tools
– Sample endpoints in server/app/mock_api.py representing external services the agent can call.

b. Relationships and interactions:
- Client ↔ WebSocket Server: bi‑directional, streams JSON and binary audio frames.
- Server ↔ OpenAI API: HTTP(s) streaming for chat completions, function‑call invocations.
- Server ↔ Mock API: HTTP calls for tool execution.
- Client: renders server’s streamed responses, plays back TTS audio.

c. Architectural patterns & principles:
– Client-Server separation via persistent WebSocket for low‑latency streaming.
– Hooks in frontend to encapsulate cross‑cutting concerns (audio, websocket).
– Agents SDK on server to centralize conversational logic and tool orchestration.
– React component decomposition for UI modularity.

d. Key technologies/frameworks:
– Next.js, React, Tailwind CSS, WebSocket API on frontend.
– FastAPI, Uvicorn, Python Agents SDK on backend.
– OpenAI REST API as external dependency.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend (Next.js Client)"
        NC["Next.js Client UI"]:::frontend
        AC["AudioChat Component"]:::frontend
        CD["ChatDialog Component"]:::frontend
        CP["Composer Component"]:::frontend
        AP["AudioPlayback Component"]:::frontend
        MB["MessageBubble Components"]:::frontend
        WH["useWebsocket Hook"]:::frontend
        AU["useAudio Hook"]:::frontend
        TC["Tailwind CSS Config"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend (FastAPI Server)"
        SE["server.py (Entry Point)"]:::backend
        AI["app/__init__.py (App Init & Routes)"]:::backend
        WS["WebSocket Handler & Streaming"]:::backend
        SD["Agents SDK Core"]:::backend
        ACfg["agent_config.py"]:::backend
        MAPI["mock_api.py (Tool Adapter)"]:::backend
        MK["Makefile (Orchestration)"]:::backend
        ENV["Env Vars (OPENAI_API_KEY)"]:::config
    end

    %% External Services
    OpenAI["OpenAI API"]:::external

    %% Connections
    NC -.->|"audio & text events"| WS
    WS -->|"streamed responses"| NC
    WS -->|"route to agent logic"| SD
    SD -->|"HTTP streaming (chat & function calls)"| OpenAI
    SD -->|"function/tool call"| MAPI

    %% Click Events
    click NC "https://github.com/openai/openai-voice-agent-sdk-sample/tree/main/frontend/src/"
    click AC "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/components/AudioChat.tsx"
    click CD "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/components/ChatDialog.tsx"
    click CP "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/components/Composer.tsx"
    click AP "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/components/AudioPlayback.tsx"
    click MB "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/components/MessageBubble.tsx"
    click MB "https://github.com/openai/openai-voice-agent-sdk-sample/tree/main/frontend/src/components/messages/"
    click WH "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/hooks/useWebsocket.ts"
    click AU "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/hooks/useAudio.ts"
    click TC "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/tailwind.config.mjs"
    click TC "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/frontend/src/app/globals.css"
    click SE "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/server/server.py"
    click AI "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/server/app/__init__.py"
    click WS "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/server/app/utils.py"
    click ACfg "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/server/app/agent_config.py"
    click MAPI "https://github.com/openai/openai-voice-agent-sdk-sample/blob/main/server/app/mock_api.py"
    click MK "https://github.com/openai/openai-voice-agent-sdk-sample/tree/main/Makefile"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085
    classDef backend fill:#d4edda,stroke:#155724,color:#155724
    classDef external fill:#ffeeba,stroke:#856404,color:#856404
    classDef config fill:#f8d7da,stroke:#721c24,color:#721c24
```
