How datascale-ai/opentalking works
OpenTalking is a real-time digital-human conversation and media-production framework. Its principal flow is browser interaction, session creation and task dispatch, speech/agent processing, avatar rendering, then WebRTC playback or external streaming. The map also preserves memory/knowledge, avatar and voice assets, video creation/cloning, and output management. Some subsystem links reflect README-described capabilities and tree-supported modules; unsampled implementation details are not asserted as specific calls.
Main entry points: main.py REST API · runner.py Session runner · context_builder.py Agent context · render_pipeline.py Speech rendering
Components 22
Interaction and API
- Conversation WebUI
apps/web/src/App.tsx - REST API
apps/api/main.py - Session routes
apps/api/routes/sessions.py
Session orchestration
- Session service
apps/api/services/session_service.py - Task broker
opentalking/core/bus.py - Task consumer
opentalking/runtime/task_consumer.py - Session runner
opentalking/pipeline/session/runner.py
Conversation services
- Agent context
opentalking/agent/context_builder.py - LLM providers
opentalking/providers/llm/openai_compatible/adapter.py - Speech recognition
opentalking/providers/stt/factory.py - Memory and knowledge
opentalking/providers/memory/runtime.py - Speech synthesis
opentalking/providers/tts/factory.py
Rendering and delivery
- Speech rendering
opentalking/pipeline/speak/render_pipeline.py - Avatar model adapters
opentalking/models/registry.py - Media fan-out
opentalking/streaming/manager.py - WebRTC playback
opentalking/providers/rtc/aiortc/adapter.py - Live outputs
opentalking/streaming/destinations/rtmps.py
Assets and production
- Avatar and voice assets
opentalking/avatar/loader.py - Video creation
opentalking/video_creation.py - Video cloning
opentalking/providers/synthesis/video_clone/ws_client.py - Persona packages
opentalking/persona/package.py
Other components
- User
Connections 23
- User → Conversation WebUI: interacts
- Conversation WebUI → REST API: HTTP requests
- REST API → Session routes: routes requests
- Session routes → Session service: manages sessions
- Session service → Task broker: dispatches tasks
- Task broker → Task consumer: delivers tasks
- Task consumer → Session runner: runs session
- Session runner → Agent context: builds context
- Agent context → Memory and knowledge: retrieves memory
- Agent context → LLM providers: requests replies
- Session runner → Speech recognition: transcribes input
- Session runner → Speech synthesis: synthesizes speech
- Session runner → Speech rendering: renders speech
- Speech rendering → Avatar model adapters: infers frames
- Avatar model adapters → Avatar and voice assets: uses avatar assets
- Speech rendering → Media fan-out: emits media
- Media fan-out → WebRTC playback: delivers playback
- WebRTC playback → Conversation WebUI: plays streams
- Media fan-out → Live outputs: fans out media
- REST API → Video creation: exposes creation
- REST API → Video cloning: exposes cloning
- REST API → Persona packages: manages personas
- REST API → Memory and knowledge: manages memory