How meeting-baas/speaking-meeting-bot works
Project type and purpose
• This is an API-first, backend-driven “meeting bot” system, built on top of the Pipecat real-time audio framework.
• Purpose: spawn AI agents (with custom Markdown-defined personas) that join Google Meet or Teams calls via MeetingBaas, process audio via WebSockets, and speak/respond using STT→LLM→TTS pipelines.
High-level file-structure analysis
• app/: FastAPI HTTP & WebSocket server handlers (routes.py, websockets.py, main.py).
• core/: Pipecat integration logic—audio frame conversion, routing, process pipelines.
• protobufs/: gRPC frames definition for audio payloads.
• config/: persona Markdown definitions, CLI scripts for persona/image/id generation, ngrok config.
• meetingbaas_pipecat/: MeetingBaas client utilities (logging, helper functions).
• scripts/: standalone CLI entrypoints (meetingbaas.py, meetingbaas_api.py).
• utils/: helper modules (ngrok tunnel lookup, URL, process utils).
• tests/: unit tests.
• Dockerfile, fly.toml, .github workflows for CI/CD, environment templates.
a. MeetingBaas Service (external)
• Video-meeting orchestrator.
• Exposes REST APIs to create bots and a WebSocket endpoint for audio streams.
b. FastAPI Server
• HTTP endpoints:
• GET / → health check
• POST /run-bots → spawn MeetingBaas bots
• WebSocket endpoint: /ws/{client_id} ←<--> MeetingBaas for raw audio.
• Pipecat WS endpoint: /pipecat/{client_id} ←<--> Pipecat core for audio frame conversion.
c. Pipecat Core (core/)
• Audio processing pipeline:
• Voice activity detection
• gRPC frame serialization/deserialization
• Message context management
• Routes audio frames between WebSocket and converter.
d. External AI Services
• STT: Deepgram or Gladia
• LLM: OpenAI GPT-4
• TTS: Cartesia
• (Image gen via Replicate, hosting via UploadThing in persona CLI flows)
e. Persona Store (config/personas/)
• Markdown definitions loaded at runtime by image_service/persona_utils.
• Provides personality context to LLM prompts.
f. CLI & Utilities
• Scripts for persona/image generation and ngrok auto-configuration in local dev.
Relationships & data flows
1. Client issues POST /run-bots → FastAPI
2. FastAPI calls MeetingBaas REST API → MeetingBaas returns bot_id & WS URL
3. MeetingBaas service connects to FastAPI /ws/{client_id}
4. FastAPI forwards binary audio frames to Pipecat WS /pipecat/{client_id}
5. Pipecat Core:
• STT service converts audio→text
• LLM service processes text with persona context → reply text
• TTS service converts reply text→audio frames
6. Pipecat → FastAPI → MeetingBaas WS → meeting participants
7. On demand: FastAPI’s image_service calls Replicate & UploadThing for avatar images
Architectural patterns & design principles
• API-First: clear separation of HTTP control plane (bot management) vs. WebSocket data plane (audio streams).
• Modular: distinct layers (app / core / services / utils) follow single-responsibility.
• Typed contracts: Pydantic models for all API requests/responses.
• Plugin-style personas: Markdown files as extensible behavior modules.
• Fallback layering for WebSocket URL resolution (config/env/ngrok/headers).