# meeting-baas/speaking-meeting-bot architecture

> Architecture diagram of the GitHub repository meeting-baas/speaking-meeting-bot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meeting-baas/speaking-meeting-bot
- Repository: https://github.com/meeting-baas/speaking-meeting-bot

## Overview

##### Project type and purpose
- This is an API-first, backend-driven “meeting bot” system, built on top of the Pipecat real-time audio framework.
- Purpose: spawn AI agents (with custom Markdown-defined personas) that join Google Meet or Teams calls via MeetingBaas, process audio via WebSockets, and speak/respond using STT→LLM→TTS pipelines.

##### High-level file-structure analysis
- app/: FastAPI HTTP & WebSocket server handlers (routes.py, websockets.py, main.py).
- core/: Pipecat integration logic—audio frame conversion, routing, process pipelines.
- protobufs/: gRPC frames definition for audio payloads.
- config/: persona Markdown definitions, CLI scripts for persona/image/id generation, ngrok config.
- meetingbaas_pipecat/: MeetingBaas client utilities (logging, helper functions).
- scripts/: standalone CLI entrypoints (meetingbaas.py, meetingbaas_api.py).
- utils/: helper modules (ngrok tunnel lookup, URL, process utils).
- tests/: unit tests.
- Dockerfile, fly.toml, .github workflows for CI/CD, environment templates.

a. MeetingBaas Service (external)
- Video-meeting orchestrator.
- Exposes REST APIs to create bots and a WebSocket endpoint for audio streams.
b. FastAPI Server
- HTTP endpoints:
- GET / → health check
- POST /run-bots → spawn MeetingBaas bots
- WebSocket endpoint: /ws/{client_id} ←<--> MeetingBaas for raw audio.
- Pipecat WS endpoint: /pipecat/{client_id} ←<--> Pipecat core for audio frame conversion.
c. Pipecat Core (core/)
- Audio processing pipeline:
- Voice activity detection
- gRPC frame serialization/deserialization
- Message context management
- Routes audio frames between WebSocket and converter.
d. External AI Services
- STT: Deepgram or Gladia
- LLM: OpenAI GPT-4
- TTS: Cartesia
- (Image gen via Replicate, hosting via UploadThing in persona CLI flows)
e. Persona Store (config/personas/)
- Markdown definitions loaded at runtime by image_service/persona_utils.
- Provides personality context to LLM prompts.
f. CLI & Utilities
- Scripts for persona/image generation and ngrok auto-configuration in local dev.

##### Relationships & data flows
1. Client issues POST /run-bots → FastAPI
2. FastAPI calls MeetingBaas REST API → MeetingBaas returns bot_id & WS URL
3. MeetingBaas service connects to FastAPI /ws/{client_id}
4. FastAPI forwards binary audio frames to Pipecat WS /pipecat/{client_id}
5. Pipecat Core:
- STT service converts audio→text
- LLM service processes text with persona context → reply text
- TTS service converts reply text→audio frames
6. Pipecat → FastAPI → MeetingBaas WS → meeting participants
7. On demand: FastAPI’s image_service calls Replicate & UploadThing for avatar images

##### Architectural patterns & design principles
- API-First: clear separation of HTTP control plane (bot management) vs. WebSocket data plane (audio streams).
- Modular: distinct layers (app / core / services / utils) follow single-responsibility.
- Typed contracts: Pydantic models for all API requests/responses.
- Plugin-style personas: Markdown files as extensible behavior modules.
- Fallback layering for WebSocket URL resolution (config/env/ngrok/headers).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Control Plane (HTTP)
    subgraph "Control Plane (HTTP)"
        direction TB
        Main["app/main.py"]:::app
        Routes["app/routes.py"]:::app
        Models["app/models.py"]:::app
        ImageSvc["app/services/image_service.py"]:::app
    end

    %% Data Plane (WebSocket)
    subgraph "Data Plane (WebSocket)"
        direction TB
        WSIn["WebSocket /ws/{client_id}"]:::app
        PipecatWS["WebSocket /pipecat/{client_id}"]:::app
    end

    %% Processing Plane (Pipecat Core)
    subgraph "Processing Plane (Pipecat Core)"
        direction TB
        Conn["core/connection.py"]:::core
        Router["core/router.py"]:::core
        Converter["core/converter.py"]:::core
        Process["core/process.py"]:::core
    end

    %% External Services
    subgraph "External Services"
        direction TB
        MB["MeetingBaas API & WS"]:::external
        STT["STT (Deepgram/Gladia)"]:::external
        LLM["LLM (OpenAI GPT-4)"]:::external
        TTS["TTS (Cartesia)"]:::external
        Replicate["Replicate (Image Gen)"]:::external
        UploadThing["UploadThing"]:::external
    end

    %% Persona Store & Context Loader
    subgraph "Persona Store & CLI"
        direction TB
        PersonaUtils["config/persona_utils.py"]:::config
        Prompts["config/prompts.py"]:::config
        Personas["config/personas/"]:::config
        CreatePersona["config/create_persona.py"]:::config
        GenerateImages["config/generate_images.py"]:::config
        ImageUploader["config/image_uploader.py"]:::config
        Migrate["config/migrate_personas.py"]:::config
        NgrokConfig["config/ngrok/config.yml"]:::config
        NgrokUtil["utils/ngrok.py"]:::config
        URLUtil["utils/url.py"]:::config
        ProcUtil["utils/process.py"]:::config
        ScriptsMB["scripts/meetingbaas.py"]:::config
        ScriptsAPI["scripts/meetingbaas_api.py"]:::config
    end

    %% Protobuf Definitions
    FramesProto["protobufs/frames.proto"]:::config
    FramesPb2["protobufs/frames_pb2.py"]:::config

    %% MeetingBaas Client Utilities
    MBInit["meetingbaas_pipecat/__init__.py"]:::config
    Logger["meetingbaas_pipecat/utils/logger.py"]:::config

    %% Flows
    User -->|POST /run-bots| Routes
    Routes -->|calls REST API| MB
    MB -->|connects WS audio| WSIn
    WSIn --> PipecatWS
    PipecatWS --> Conn
    Conn --> Router
    Router --> Converter
    Converter -->|STT request| STT
    STT -->|text| Process
    PersonaUtils --> Process
    Prompts --> Process
    Personas --> Process
    Process -->|LLM request| LLM
    LLM -->|response| Process
    Process -->|TTS request| TTS
    TTS -->|audio frames| Converter
    Converter --> Router
    Router --> Conn
    Conn --> WSIn
    WSIn -->|audio WS| MB
    ImageSvc -->|image gen request| Replicate
    Replicate --> UploadThing
    ImageSvc -->|avatar upload| UploadThing

    %% Additional Utilities
    NgrokUtil -.-> WSIn
    NgrokConfig -.-> NgrokUtil
    URLUtil -.-> Routes
    ProcUtil -.-> Main

    %% Click Events
    click Main "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/app/main.py"
    click Routes "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/app/routes.py"
    click Models "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/app/models.py"
    click ImageSvc "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/app/services/image_service.py"
    click Conn "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/core/connection.py"
    click Converter "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/core/converter.py"
    click Process "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/core/process.py"
    click Router "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/core/router.py"
    click Logger "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/meetingbaas_pipecat/utils/logger.py"
    click MBInit "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/meetingbaas_pipecat/__init__.py"
    click FramesProto "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/protobufs/frames.proto"
    click FramesPb2 "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/protobufs/frames_pb2.py"
    click PersonaUtils "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/persona_utils.py"
    click Prompts "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/prompts.py"
    click Personas "https://github.com/meeting-baas/speaking-meeting-bot/tree/main/config/personas/"
    click CreatePersona "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/create_persona.py"
    click GenerateImages "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/generate_images.py"
    click ImageUploader "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/image_uploader.py"
    click Migrate "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/migrate_personas.py"
    click NgrokConfig "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/config/ngrok/config.yml"
    click NgrokUtil "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/utils/ngrok.py"
    click URLUtil "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/utils/url.py"
    click ProcUtil "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/utils/process.py"
    click ScriptsMB "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/scripts/meetingbaas.py"
    click ScriptsAPI "https://github.com/meeting-baas/speaking-meeting-bot/blob/main/scripts/meetingbaas_api.py"

    %% Styles
    classDef app fill:#D0E8FF,stroke:#0366D6
    classDef core fill:#E0F8E0,stroke:#28A745
    classDef config fill:#FFE5B4,stroke:#FF8C00
    classDef external fill:#EEEEEE,stroke:#666666,stroke-dasharray: 2 2
```
