How pipecat-ai/pipecat works
Purpose and Entry Points
Pipecat is a Python 3.11+ framework for composing low-latency voice and multimodal AI agents. Applications primarily import the pipecat library and assemble a pipeline; the packaged CLI scaffolds projects and runs evaluations. Core entry points are src/pipecat/cli/main.py, src/pipecat/runner/run.py, and src/pipecat/evals/__main__.py.
Frame-Based Processing Model
The central data contract is typed frames: audio, text, control, LLM, and transport events move through processors rather than application-specific callbacks. FrameProcessor implementations transform, consume, or emit frames, allowing service and transport implementations to remain composable. The wire/schema boundary is represented by src/pipecat/frames/frames.py, src/pipecat/frames/frames.proto, and src/pipecat/processors/frame_processor.py.
Pipeline Execution and Concurrency
A Pipeline orders processors; PipelineTask and PipelineRunner manage lifecycle, scheduling, cancellation, and frame traversal. Parallel and synchronous-parallel variants support fan-out/fan-in, while switchers replace LLMs or services at runtime. This is the in-process agent execution boundary. Key paths: src/pipecat/pipeline/pipeline.py, src/pipecat/pipeline/task.py, src/pipecat/pipeline/runner.py.
Conversational Audio Path
A typical cascaded voice agent is: transport input audio → filtering/VAD and turn control → STT → LLM context/response aggregation → TTS → transport output. Audio utilities include resampling, mixers, noise filters, VAD, and local ONNX/CoreML turn models. Turn strategies explicitly decide user-turn start/stop, interruption, mute, and idle behavior rather than delegating all control to providers. Key paths: src/pipecat/processors/audio/vad_processor.py, src/pipecat/turns/user_turn_controller.py, src/pipecat/processors/aggregators/llm_context.py.
AI-Service and Tool Boundary
Abstract AI service bases normalize provider behavior across STT, TTS, LLM, vision/image, MCP, and realtime speech-to-speech services. Provider packages implement concrete HTTP/WebSocket SDK integrations; adapters reconcile provider-specific tool/function schemas and context representations. Important contracts and examples: src/pipecat/services/ai_service.py, src/pipecat/services/llm_service.py, src/pipecat/adapters/services/open_ai_adapter.py. External dependencies include OpenAI, Anthropic, AWS/Bedrock/Nova Sonic, Google/Gemini, Azure, Deepgram, ElevenLabs, and many others; optional extras keep the base runtime lightweight.
Transport, Telephony, and Serialization
Transports isolate media/session ingress and egress from pipeline logic. Supported boundaries include local devices, WebSockets/FastAPI, WebRTC platforms (Daily, LiveKit, MoQ, Vonage), WhatsApp, and video-avatar transports. Telephony providers are handled through serializer modules that translate provider payloads to/from frames. Key paths: src/pipecat/transports/base_transport.py, src/pipecat/transports/websocket/fastapi.py, src/pipecat/serializers/twilio.py.
Multi-Agent Coordination and Structured Flows
Each pipeline can act as an agent. The bus supports local async queues and distributed Redis or PGMQ backends, with bridge processors adapting messages into pipeline flow. Workers provide LLM, proxy, and UI roles. Separately, Flows provides stateful, structured conversation actions over LLM/tool interactions. Key paths: src/pipecat/bus/bus.py, src/pipecat/bus/network/redis.py, src/pipecat/flows/manager.py.
Operations, Observability, and Delivery
Metrics, observers, tracing, and Sentry integration observe frame/turn latency and service behavior without becoming business logic. Evaluation scenarios execute through a dedicated harness. The project uses Python packaging with uv and optional extras (pyproject.toml), while generated CLI templates can target Dockerized server/client deployments. Key paths: src/pipecat/metrics/metrics.py, src/pipecat/utils/tracing/setup.py, src/pipecat/evals/harness.py.
Main entry points: frames.py Typed frames · llm_service.py LLM service base · pipeline.py Pipeline · runner.py Pipeline runner
Components 24
Entry Points
- CLI · Scaffolds projects and runs evaluation commands.
src/pipecat/cli/main.py - Runner entry
src/pipecat/runner/run.py - Eval entry
src/pipecat/evals/__main__.py
Pipeline Runtime
In-process frame execution and conversational control.
- Typed frames · Audio, text, control, LLM, and transport events.
src/pipecat/frames/frames.py - Frame schema
src/pipecat/frames/frames.proto - Frame processor · Composable transform, consume, and emit boundary.
src/pipecat/processors/frame_processor.py - Pipeline · Orders processors, including parallel and switching variants.
src/pipecat/pipeline/pipeline.py - Pipeline task
src/pipecat/pipeline/task.py - Pipeline runner · Runs tasks with lifecycle, cancellation, and traversal management.
src/pipecat/pipeline/runner.py - VAD processor · Filters speech activity for turn handling.
src/pipecat/processors/audio/vad_processor.py - User turn control · Controls starts, stops, interruptions, mute, and idle behavior.
src/pipecat/turns/user_turn_controller.py - LLM context · Aggregates conversation context and responses.
src/pipecat/processors/aggregators/llm_context.py
Integrations
Provider and media boundaries.
- AI service base · Normalizes STT, TTS, vision, realtime, and MCP providers.
src/pipecat/services/ai_service.py - LLM service base
src/pipecat/services/llm_service.py - OpenAI adapter · Reconciles provider tool and context schemas.
src/pipecat/adapters/services/open_ai_adapter.py - Base transport · Isolates transport ingress and egress from pipeline logic.
src/pipecat/transports/base_transport.py - FastAPI WebSocket
src/pipecat/transports/websocket/fastapi.py - Twilio serializer · Translates provider payloads to and from frames.
src/pipecat/serializers/twilio.py
Coordination & Operations
- Agent bus · Connects pipeline agents through local or distributed messaging.
src/pipecat/bus/bus.py - Redis backend
src/pipecat/bus/network/redis.py - Flows manager · Coordinates stateful conversation actions over LLM and tools.
src/pipecat/flows/manager.py - Metrics · Captures frame, turn, and service latency.
src/pipecat/metrics/metrics.py - Tracing setup
src/pipecat/utils/tracing/setup.py - Evaluation harness · Executes evaluation scenarios.
src/pipecat/evals/harness.py
Connections 22
- CLI → Runner entry: starts apps
- CLI → Eval entry: runs evals
- Eval entry → Evaluation harness: executes
- Frame schema → Typed frames: defines
- Typed frames → Frame processor: flows through
- Frame processor → Pipeline: composes into
- Pipeline → Pipeline task: is managed by
- Pipeline task → Pipeline runner: scheduled by
- Base transport → Typed frames: ingress and egress
- FastAPI WebSocket → Base transport: implements
- Twilio serializer → Typed frames: serializes
- VAD processor → User turn control: speech activity
- User turn control → LLM context: turn events
- LLM context → LLM service base: LLM requests
- LLM service base → AI service base: specializes
- OpenAI adapter → LLM service base: adapts
- Agent bus → Pipeline: bridges agents
- Redis backend → Agent bus: distributed backend
- Flows manager → LLM service base: orchestrates tools
- Metrics → Pipeline runner: observes
- Tracing setup → AI service base: traces
- Evaluation harness → Pipeline runner: evaluates