# google-gemini/live-api-web-console architecture

> Architecture diagram of the GitHub repository google-gemini/live-api-web-console, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-gemini/live-api-web-console
- Repository: https://github.com/google-gemini/live-api-web-console

## Overview

##### Project type and purpose
- This is a single-page React application (bootstrapped with Create React App) that serves as a Web Console for Google’s Live API.
- It runs entirely in the browser, captures user media (microphone, webcam, screen), streams audio via WebSocket to the Live API, and renders responses (including function calls for graph rendering).

##### File structure
- Top-level artifacts:
- public/ – static HTML, robots.txt
- src/ – React code: components/, contexts/, hooks/, lib/, App.tsx, index.tsx, styles
- app.yaml – configuration for deploying static assets (e.g., on Google App Engine)
- .env – holds the Gemini API key
- Patterns:
- Context API (LiveAPIContext) to supply the WebSocket client and config across components
- Custom hooks (use-live-api, use-media-stream-mux, use-webcam, use-screen-capture) abstract media capture and streaming logic
- lib/worklets – Web Audio API AudioWorklets for real-time audio processing (volume meter, audio processing)
- Dedicated UI modules: Altair (graph renderer), AudioPulse (audio level visualization), ControlTray (mic/webcam controls), Logger (event log viewer), SettingsDialog, SidePanel

##### README for additional insights
- The app’s core functions:
- Record and stream user media (audio, video)
- Listen for responses and “toolcall” events
- Provide a developer-friendly log view
- Render custom components (e.g., vega graphs via Altair) upon function calls

a. Main components
- Browser / React SPA
- UI Components (ControlTray, SidePanel, Logger, SettingsDialog, Altair, AudioPulse)
- React Context Layer (LiveAPIContext)
- Custom Hooks (media capture, streaming abstraction)
- AudioWorklet processors (audio-processing.ts, vol-meter.ts)
- WebSocket Client (genai-live-client.ts)
- External Live API WebSocket server (Google Live API endpoint)
- Deployment layer (static host / App Engine)
b. Relationships & data flow
1. User interacts with UI (e.g., toggles mic) → ControlTray
2. MediaCapture hooks acquire MediaStream (microphone, webcam, screen)
3. AudioWorklets process raw audio → volume meter + audio frames
4. WebSocket Client streams audio frames to Live API
5. Live API returns responses or toolcall events → Client emits events
6. LiveAPIContext propagates events → React components subscribe (Logger, Altair)
7. Altair receives functionCall → vega-embed renders graphs
c. Architectural patterns / design principles
- Separation of concerns: UI vs. streaming logic vs. processing logic
- Event-driven architecture: centralized EventEmitter in WebSocket client
- Context + custom hooks for state & side-effect management
- Modular Worklets for high-performance audio processing
d. Key technologies & libraries
- React, TypeScript, Create React App
- React Context API, custom hooks
- Web Audio API & AudioWorklets
- WebSockets (genai-live-client)
- Vega-embed for chart rendering
- Google Live API (Gemini) over WebSocket
- CSS/SCSS modules for styling

## Diagram (Mermaid)

```mermaid
graph TB
    %% Browser Runtime
    subgraph "Browser (Client-Side)"
        direction TB

        %% Entry Points
        Index["src/index.tsx\n&\nsrc/App.tsx"]:::logic
        click Index "https://github.com/google-gemini/live-api-web-console/blob/main/src/index.tsx"
        click Index "https://github.com/google-gemini/live-api-web-console/blob/main/src/App.tsx"

        %% Context & Hooks Layer
        subgraph "Context & Hooks" 
            direction TB
            LiveAPIContext["LiveAPIContext"]:::logic
            click LiveAPIContext "https://github.com/google-gemini/live-api-web-console/blob/main/src/contexts/LiveAPIContext.tsx"
            UseLiveAPI["use-live-api"]:::logic
            click UseLiveAPI "https://github.com/google-gemini/live-api-web-console/blob/main/src/hooks/use-live-api.ts"
            UseMediaMux["use-media-stream-mux"]:::logic
            click UseMediaMux "https://github.com/google-gemini/live-api-web-console/blob/main/src/hooks/use-media-stream-mux.ts"
            UseWebcam["use-webcam"]:::logic
            click UseWebcam "https://github.com/google-gemini/live-api-web-console/blob/main/src/hooks/use-webcam.ts"
            UseScreen["use-screen-capture"]:::logic
            click UseScreen "https://github.com/google-gemini/live-api-web-console/blob/main/src/hooks/use-screen-capture.ts"
        end

        %% UI Components
        subgraph "UI Components" 
            direction TB
            ControlTray["ControlTray"]:::ui
            click ControlTray "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/control-tray/ControlTray.tsx"
            SidePanel["SidePanel"]:::ui
            click SidePanel "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/side-panel/SidePanel.tsx"
            LoggerComp["Logger"]:::ui
            click LoggerComp "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/logger/Logger.tsx"
            SettingsDialog["SettingsDialog"]:::ui
            click SettingsDialog "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/settings-dialog/SettingsDialog.tsx"
            ResponseModality["ResponseModalitySelector"]:::ui
            click ResponseModality "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/settings-dialog/ResponseModalitySelector.tsx"
            VoiceSelector["VoiceSelector"]:::ui
            click VoiceSelector "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/settings-dialog/VoiceSelector.tsx"
            AltairComp["Altair"]:::ui
            click AltairComp "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/altair/Altair.tsx"
            AudioPulse["AudioPulse"]:::ui
            click AudioPulse "https://github.com/google-gemini/live-api-web-console/blob/main/src/components/audio-pulse/AudioPulse.tsx"
        end

        %% Lib & Worklets
        subgraph "Logic & Processing" 
            direction TB
            GenAIClient["genai-live-client"]:::logic
            click GenAIClient "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/genai-live-client.ts"
            AudioRecorder["audio-recorder"]:::logic
            click AudioRecorder "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/audio-recorder.ts"
            AudioStreamer["audio-streamer"]:::logic
            click AudioStreamer "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/audio-streamer.ts"
            WorkletRegistry["audioworklet-registry"]:::logic
            click WorkletRegistry "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/audioworklet-registry.ts"
            StoreLogger["store-logger"]:::logic
            click StoreLogger "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/store-logger.ts"
            Utils["utils"]:::logic
            click Utils "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/utils.ts"
            subgraph "AudioWorklets" 
                direction TB
                AudioProc["audio-processing"]:::worklet
                click AudioProc "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/worklets/audio-processing.ts"
                VolMeter["vol-meter"]:::worklet
                click VolMeter "https://github.com/google-gemini/live-api-web-console/blob/main/src/lib/worklets/vol-meter.ts"
            end
        end
    end

    %% External Service
    LiveAPI["Google Live API (WebSocket)"]:::external

    %% Connections
    Index -->|renders| LiveAPIContext
    LiveAPIContext -->|provides context to| ControlTray
    LiveAPIContext -->|provides context to| SidePanel
    LiveAPIContext -->|provides context to| LoggerComp
    LiveAPIContext -->|provides context to| SettingsDialog
    SettingsDialog -->|includes| ResponseModality
    SettingsDialog -->|includes| VoiceSelector
    LiveAPIContext -->|provides context to| AltairComp
    LiveAPIContext -->|provides context to| AudioPulse

    ControlTray -->|uses hooks| UseMediaMux
    ControlTray -->|uses hooks| UseWebcam
    ControlTray -->|uses hooks| UseScreen

    UseMediaMux --> AudioRecorder
    AudioRecorder --> AudioStreamer
    AudioStreamer --> GenAIClient

    UseLiveAPI --> GenAIClient
    LiveAPIContext --> UseLiveAPI

    WorkletRegistry --> AudioProc
    WorkletRegistry --> VolMeter
    AudioProc --> AudioStreamer
    VolMeter --> AudioPulse

    GenAIClient -->|streams audio frames via WS| LiveAPI
    LiveAPI -->|emits responses/events| GenAIClient
    GenAIClient -->|emits events| LiveAPIContext

    LiveAPIContext -->|logs events| StoreLogger
    StoreLogger --> LoggerComp

    LiveAPIContext -->|invokes functionCall for graphs| AltairComp

    %% Styling
    classDef ui fill:#D0E6FF,stroke:#0366D6,color:#0366D6;
    classDef logic fill:#DFFFE0,stroke:#28A745,color:#28A745;
    classDef worklet fill:#FFEFD5,stroke:#FF8C00,color:#FF8C00;
    classDef external fill:#FFDDDD,stroke:#D73A49,color:#D73A49;
```
