# heiko-hotz/gemini-multimodal-live-dev-guide architecture

> Architecture diagram of the GitHub repository heiko-hotz/gemini-multimodal-live-dev-guide, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/heiko-hotz/gemini-multimodal-live-dev-guide
- Repository: https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide

## Overview

##### High-Level Components

a) Introduction Layer (Part 1)
- SDK Integration
- Basic Audio Processing
- Initial Setup Components

b) Development API Implementation (Part 2)
- WebSocket Communication
- Audio/Video Processing
- Function Calling System
- Tools Integration

c) Vertex AI Implementation (Part 3)
- Enterprise Architecture
- Proxy Layer
- Security Components

##### Core Components

Frontend Components:
- Audio Processing Module
- AudioWorklet System
- Recording Handler
- Streaming Components
- Video Processing Module
- Media Handler
- UI Components
- WebSocket Client

Backend Components:
- Python Server
- Proxy Service (for Vertex AI)
- Authentication Layer
- API Integration Layer

External Services:
- Gemini Multimodal Live API
- Weather API
- Stock API
- Vertex AI (for Part 3)

##### Key Relationships to Highlight

- Client and WebSocket server
- Audio/Video processors and Media handlers
- Proxy and Vertex AI
- Function calling system and external tools

##### Technical Details

Media Processing:
- Audio chunking
- VAD (Voice Activity Detection)
- Frame processing
- Real-time streaming

Security Layer:
- API key management
- Service account authentication
- Proxy configuration

##### Specific Implementation Notes

- Audio processing pipeline
- Video processing pipeline
- WebSocket communication flow
- Function calling architecture

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client_Layer
        UI["Frontend UI"]:::frontend
        click UI "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/part_2_dev_api/chapter_08/index.html"
        AudioProc["Audio Processing Module"]:::media
        click AudioProc "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/audio-recording-worklet.js"
        VideoProc["Video Processing Module"]:::media
        MediaHandler["Media Handler"]:::media
        click MediaHandler "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/media-handler.js"
        StatusMgr["Status Management"]:::frontend
        click StatusMgr "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/part_2_dev_api/chapter_08/status-handler.js"
    end

    subgraph Processing_Layer
        AudioWorklet["AudioWorklet System"]:::media
        click AudioWorklet "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/audioworklet-registry.js"
        RecHandler["Recording Handler"]:::media
        click RecHandler "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/audio-recorder.js"
        StreamComp["Streaming Components"]:::media
        click StreamComp "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/audio-streamer.js"
        WSClient["WebSocket Client"]:::network
    end

    subgraph Backend_Layer
        PyServer["Python Server"]:::backend
        click PyServer "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/server.py"
        ProxyService["Proxy Service"]:::backend
        click ProxyService "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/part_3_vertex_api/proxy/proxy.py"
        AuthLayer["Authentication Layer"]:::security
        APIIntegration["API Integration Layer"]:::backend
        click APIIntegration "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/gemini-live-api.js"
    end

    subgraph External_Services
        GeminiAPI["Gemini Multimodal Live API"]:::external
        WeatherAPI["Weather API"]:::external
        click WeatherAPI "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/tools/weather-api.js"
        StockAPI["Stock API"]:::external
        click StockAPI "https://github.com/heiko-hotz/gemini-multimodal-live-dev-guide/blob/main/shared/tools/stock-api.js"
        VertexAI["Vertex AI"]:::external
    end

    %% Connections
    UI --> AudioProc
    UI --> VideoProc
    UI --> MediaHandler
    UI --> StatusMgr

    AudioProc --> AudioWorklet
    AudioProc --> RecHandler
    AudioProc --> StreamComp
    VideoProc --> MediaHandler
    MediaHandler --> WSClient

    WSClient --> PyServer
    PyServer --> AuthLayer
    PyServer --> APIIntegration
    ProxyService --> VertexAI
    APIIntegration --> GeminiAPI
    APIIntegration --> WeatherAPI
    APIIntegration --> StockAPI

    %% Styles
    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef media fill:#9b59b6,stroke:#8e44ad,color:white
    classDef network fill:#2ecc71,stroke:#27ae60,color:white
    classDef backend fill:#16a085,stroke:#1abc9c,color:white
    classDef security fill:#e74c3c,stroke:#c0392b,color:white
    classDef external fill:#f1c40f,stroke:#f39c12,color:black

    %% Legend
    subgraph Legend
        Frontend["Frontend Component"]:::frontend
        Media["Media Processing"]:::media
        Network["Network Component"]:::network
        Backend["Backend Service"]:::backend
        Security["Security Component"]:::security
        External["External Service"]:::external
    end
```
