How livekit/client-sdk-esp32 works
Main Components
• Application Layer (Examples)
• minimal, minimal_video, voice_agent, etc.
• Uses livekit_system_init(), configures capturer & renderer, creates/connects livekit_room.
• Example Utilities (components/example_utils)
• Helper functions for networking, logging, JSON, token handling.
• LiveKit Core Component (components/livekit)
Subcomponents:
• System – initialization, event loop.
• Engine – media pipeline orchestration.
• Signaling – WebSocket/HTTP handshake with LiveKit server.
• Protocol – Encode/decode protobuf messages.
• RPC Manager – remote method calls.
• Peer Manager – tracks participants & their media streams.
• Protocol Definitions (components/livekit/protocol)
• .proto files → generated C via Nanopb.
• Token Management (components/sandbox_token)
• Generates or manages JWT tokens for auth (in sandbox mode).
• Third‐Party Libraries (components/third_party)
• Nanopb – lightweight Protobuf C implementation.
• Khash – hash‐map utility for data structures.
• Hardware Drivers & IDF Components
• esp_capture – camera/mic capture (I2S, MIPI CSI).
• av_render – audio/video output (I2S, LCD).
• Wi-Fi / TCP/IP stack of ESP-IDF.
• External Service
• LiveKit Server (cloud or self‐hosted) – signaling & SFU for media routing.
Relationships & Data Flows
• Application ⇄ Example Utils: calls helper APIs (e.g., JSON, token creation).
• Application → LiveKit Core:
• livekit_system_init(), livekit_room_create(), livekit_room_connect().
• LiveKit Core → Signaling → TCP/IP → LiveKit Server
• LiveKit Core → Protocol (Nanopb) → encode/decode messages
• LiveKit Core → RPC Manager → invoke handlers or send RPCs
• LiveKit Core ↔ Engine → manage media tracks
• Engine → esp_capture (audio/video capture) → media frames
• Engine → codecs (H.264, Opus) → encoded frames
• Engine → TCP/IP → LiveKit Server
• LiveKit Server → TCP/IP → Engine → av_render → hardware outputs
• Sandbox Token component: optional path App → sandbox_token → LiveKit Server auth
Architectural Patterns & Principles
• Modular IDF Components: each feature in its own CMake/Kconfig.projbuild component.
• Layered Design:
• Hardware Drivers → Media Engine → Protocol/Signaling → Application API.
• Event-Driven: callbacks (on_state_changed, on_participant_info, RPC handlers).
• Plugin Hooks: user provides capturer/renderer handles.
Relevant Technologies, Frameworks & Libraries
• ESP-IDF build system (CMake, Kconfig)
• Nanopb (protobuf C)
• Khash
• Opus & H.264 codecs via ESP-IDF components
• WebSocket/HTTP stack of ESP-IDF
• esp_capture, av_render
• C standard library, lwIP
Elements
• Media data flow (camera → Engine → network → SFU → Engine → display)
• Control/signaling flow (App API → Signaling → Server)
• RPC calls (Engine ↔ RPC Manager ↔ App Handlers).
• Blue for IDF Components (esp_capture, av_render)
• Green for LiveKit Core modules
• Orange for Third‐Party libraries
• Yellow for Application layer
• Gray for External services (LiveKit Server).