How google-gemini/live-api-web-console works
Project type and purpose
• This is a single-page React application (bootstrapped with Create React App) that serves as a Web Console for Google’s Live API.
• It runs entirely in the browser, captures user media (microphone, webcam, screen), streams audio via WebSocket to the Live API, and renders responses (including function calls for graph rendering).
File structure
• Top-level artifacts:
• public/ – static HTML, robots.txt
• src/ – React code: components/, contexts/, hooks/, lib/, App.tsx, index.tsx, styles
• app.yaml – configuration for deploying static assets (e.g., on Google App Engine)
• .env – holds the Gemini API key
• Patterns:
• Context API (LiveAPIContext) to supply the WebSocket client and config across components
• Custom hooks (use-live-api, use-media-stream-mux, use-webcam, use-screen-capture) abstract media capture and streaming logic
• lib/worklets – Web Audio API AudioWorklets for real-time audio processing (volume meter, audio processing)
• Dedicated UI modules: Altair (graph renderer), AudioPulse (audio level visualization), ControlTray (mic/webcam controls), Logger (event log viewer), SettingsDialog, SidePanel
README for additional insights
• The app’s core functions:
• Record and stream user media (audio, video)
• Listen for responses and “toolcall” events
• Provide a developer-friendly log view
• Render custom components (e.g., vega graphs via Altair) upon function calls
a. Main components
• Browser / React SPA
• UI Components (ControlTray, SidePanel, Logger, SettingsDialog, Altair, AudioPulse)
• React Context Layer (LiveAPIContext)
• Custom Hooks (media capture, streaming abstraction)
• AudioWorklet processors (audio-processing.ts, vol-meter.ts)
• WebSocket Client (genai-live-client.ts)
• External Live API WebSocket server (Google Live API endpoint)
• Deployment layer (static host / App Engine)
b. Relationships & data flow
1. User interacts with UI (e.g., toggles mic) → ControlTray
2. MediaCapture hooks acquire MediaStream (microphone, webcam, screen)
3. AudioWorklets process raw audio → volume meter + audio frames
4. WebSocket Client streams audio frames to Live API
5. Live API returns responses or toolcall events → Client emits events
6. LiveAPIContext propagates events → React components subscribe (Logger, Altair)
7. Altair receives functionCall → vega-embed renders graphs
c. Architectural patterns / design principles
• Separation of concerns: UI vs. streaming logic vs. processing logic
• Event-driven architecture: centralized EventEmitter in WebSocket client
• Context + custom hooks for state & side-effect management
• Modular Worklets for high-performance audio processing
d. Key technologies & libraries
• React, TypeScript, Create React App
• React Context API, custom hooks
• Web Audio API & AudioWorklets
• WebSockets (genai-live-client)
• Vega-embed for chart rendering
• Google Live API (Gemini) over WebSocket
• CSS/SCSS modules for styling