# datascale-ai/opentalking architecture

> Architecture diagram of the GitHub repository datascale-ai/opentalking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datascale-ai/opentalking
- Repository: https://github.com/datascale-ai/opentalking

## Overview

OpenTalking is a real-time digital-human conversation and media-production framework. Its principal flow is browser interaction, session creation and task dispatch, speech/agent processing, avatar rendering, then WebRTC playback or external streaming. The map also preserves memory/knowledge, avatar and voice assets, video creation/cloning, and output management. Some subsystem links reflect README-described capabilities and tree-supported modules; unsampled implementation details are not asserted as specific calls.

Main entry points: [`apps/api/main.py`](https://github.com/datascale-ai/opentalking/blob/main/apps/api/main.py) (REST API), [`opentalking/pipeline/session/runner.py`](https://github.com/datascale-ai/opentalking/blob/main/opentalking/pipeline/session/runner.py) (Session runner), [`opentalking/agent/context_builder.py`](https://github.com/datascale-ai/opentalking/blob/main/opentalking/agent/context_builder.py) (Agent context), [`opentalking/pipeline/speak/render_pipeline.py`](https://github.com/datascale-ai/opentalking/blob/main/opentalking/pipeline/speak/render_pipeline.py) (Speech rendering)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui_api["Interaction and API"]
  node_web["Conversation WebUI<br/>[App.tsx]"]
  node_api["REST API<br/>[main.py]"]
  node_session_routes["Session routes<br/>[sessions.py]"]
end

subgraph group_orchestration["Session orchestration"]
  node_session_service["Session service<br/>[session_service.py]"]
  node_broker["Task broker<br/>[bus.py]"]
  node_worker["Task consumer<br/>[task_consumer.py]"]
  node_runner["Session runner<br/>[runner.py]"]
end

subgraph group_conversation["Conversation services"]
  node_agent["Agent context<br/>[context_builder.py]"]
  node_llm["LLM providers<br/>[adapter.py]"]
  node_stt["Speech recognition<br/>[factory.py]"]
  node_memory[("Memory and knowledge<br/>[runtime.py]")]
  node_synthesis["Speech synthesis<br/>[factory.py]"]
end

subgraph group_media["Rendering and delivery"]
  node_render["Speech rendering<br/>[render_pipeline.py]"]
  node_models["Avatar model adapters<br/>[registry.py]"]
  node_program["Media fan-out<br/>[manager.py]"]
  node_rtc["WebRTC playback<br/>[adapter.py]"]
  node_streaming["Live outputs<br/>[rtmps.py]"]
end

subgraph group_creative["Assets and production"]
  node_assets["Avatar and voice assets<br/>[loader.py]"]
  node_video_create["Video creation<br/>[video_creation.py]"]
  node_video_clone["Video cloning<br/>[ws_client.py]"]
  node_persona["Persona packages<br/>[package.py]"]
end

node_user(("User"))

node_user -->|"interacts"| node_web
node_web -->|"HTTP requests"| node_api
node_api -->|"routes requests"| node_session_routes
node_session_routes -->|"manages sessions"| node_session_service
node_session_service -->|"dispatches tasks"| node_broker
node_broker -->|"delivers tasks"| node_worker
node_worker -->|"runs session"| node_runner
node_runner -->|"builds context"| node_agent
node_agent -.->|"retrieves memory"| node_memory
node_agent -->|"requests replies"| node_llm
node_runner -->|"transcribes input"| node_stt
node_runner -->|"synthesizes speech"| node_synthesis
node_runner -->|"renders speech"| node_render
node_render -->|"infers frames"| node_models
node_models -->|"uses avatar assets"| node_assets
node_render -->|"emits media"| node_program
node_program -->|"delivers playback"| node_rtc
node_rtc -->|"plays streams"| node_web
node_program -->|"fans out media"| node_streaming
node_api -->|"exposes creation"| node_video_create
node_api -->|"exposes cloning"| node_video_clone
node_api -->|"manages personas"| node_persona
node_api -->|"manages memory"| node_memory

click node_web "https://github.com/datascale-ai/opentalking/blob/main/apps/web/src/App.tsx"
click node_api "https://github.com/datascale-ai/opentalking/blob/main/apps/api/main.py"
click node_session_routes "https://github.com/datascale-ai/opentalking/blob/main/apps/api/routes/sessions.py"
click node_session_service "https://github.com/datascale-ai/opentalking/blob/main/apps/api/services/session_service.py"
click node_broker "https://github.com/datascale-ai/opentalking/blob/main/opentalking/core/bus.py"
click node_worker "https://github.com/datascale-ai/opentalking/blob/main/opentalking/runtime/task_consumer.py"
click node_runner "https://github.com/datascale-ai/opentalking/blob/main/opentalking/pipeline/session/runner.py"
click node_agent "https://github.com/datascale-ai/opentalking/blob/main/opentalking/agent/context_builder.py"
click node_llm "https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/llm/openai_compatible/adapter.py"
click node_stt "https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/stt/factory.py"
click node_memory "https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/memory/runtime.py"
click node_synthesis "https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/tts/factory.py"
click node_render "https://github.com/datascale-ai/opentalking/blob/main/opentalking/pipeline/speak/render_pipeline.py"
click node_models "https://github.com/datascale-ai/opentalking/blob/main/opentalking/models/registry.py"
click node_program "https://github.com/datascale-ai/opentalking/blob/main/opentalking/streaming/manager.py"
click node_rtc "https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/rtc/aiortc/adapter.py"
click node_streaming "https://github.com/datascale-ai/opentalking/blob/main/opentalking/streaming/destinations/rtmps.py"
click node_assets "https://github.com/datascale-ai/opentalking/blob/main/opentalking/avatar/loader.py"
click node_video_create "https://github.com/datascale-ai/opentalking/blob/main/opentalking/video_creation.py"
click node_video_clone "https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/synthesis/video_clone/ws_client.py"
click node_persona "https://github.com/datascale-ai/opentalking/blob/main/opentalking/persona/package.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_api,node_session_routes,node_user toneBlue
class node_session_service,node_broker,node_worker,node_runner toneAmber
class node_agent,node_llm,node_stt,node_memory,node_synthesis toneMint
class node_render,node_models,node_program,node_rtc,node_streaming toneRose
class node_assets,node_video_create,node_video_clone,node_persona toneIndigo
```

## Components

### Interaction and API

- [Conversation WebUI](https://github.com/datascale-ai/opentalking/blob/main/apps/web/src/App.tsx): `apps/web/src/App.tsx`
- [REST API](https://github.com/datascale-ai/opentalking/blob/main/apps/api/main.py): `apps/api/main.py`
- [Session routes](https://github.com/datascale-ai/opentalking/blob/main/apps/api/routes/sessions.py): `apps/api/routes/sessions.py`

### Session orchestration

- [Session service](https://github.com/datascale-ai/opentalking/blob/main/apps/api/services/session_service.py): `apps/api/services/session_service.py`
- [Task broker](https://github.com/datascale-ai/opentalking/blob/main/opentalking/core/bus.py): `opentalking/core/bus.py`
- [Task consumer](https://github.com/datascale-ai/opentalking/blob/main/opentalking/runtime/task_consumer.py): `opentalking/runtime/task_consumer.py`
- [Session runner](https://github.com/datascale-ai/opentalking/blob/main/opentalking/pipeline/session/runner.py): `opentalking/pipeline/session/runner.py`

### Conversation services

- [Agent context](https://github.com/datascale-ai/opentalking/blob/main/opentalking/agent/context_builder.py): `opentalking/agent/context_builder.py`
- [LLM providers](https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/llm/openai_compatible/adapter.py): `opentalking/providers/llm/openai_compatible/adapter.py`
- [Speech recognition](https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/stt/factory.py): `opentalking/providers/stt/factory.py`
- [Memory and knowledge](https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/memory/runtime.py): `opentalking/providers/memory/runtime.py`
- [Speech synthesis](https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/tts/factory.py): `opentalking/providers/tts/factory.py`

### Rendering and delivery

- [Speech rendering](https://github.com/datascale-ai/opentalking/blob/main/opentalking/pipeline/speak/render_pipeline.py): `opentalking/pipeline/speak/render_pipeline.py`
- [Avatar model adapters](https://github.com/datascale-ai/opentalking/blob/main/opentalking/models/registry.py): `opentalking/models/registry.py`
- [Media fan-out](https://github.com/datascale-ai/opentalking/blob/main/opentalking/streaming/manager.py): `opentalking/streaming/manager.py`
- [WebRTC playback](https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/rtc/aiortc/adapter.py): `opentalking/providers/rtc/aiortc/adapter.py`
- [Live outputs](https://github.com/datascale-ai/opentalking/blob/main/opentalking/streaming/destinations/rtmps.py): `opentalking/streaming/destinations/rtmps.py`

### Assets and production

- [Avatar and voice assets](https://github.com/datascale-ai/opentalking/blob/main/opentalking/avatar/loader.py): `opentalking/avatar/loader.py`
- [Video creation](https://github.com/datascale-ai/opentalking/blob/main/opentalking/video_creation.py): `opentalking/video_creation.py`
- [Video cloning](https://github.com/datascale-ai/opentalking/blob/main/opentalking/providers/synthesis/video_clone/ws_client.py): `opentalking/providers/synthesis/video_clone/ws_client.py`
- [Persona packages](https://github.com/datascale-ai/opentalking/blob/main/opentalking/persona/package.py): `opentalking/persona/package.py`

### Other components

- **User**

## Connections

- User → Conversation WebUI: interacts
- Conversation WebUI → REST API: HTTP requests
- REST API → Session routes: routes requests
- Session routes → Session service: manages sessions
- Session service → Task broker: dispatches tasks
- Task broker → Task consumer: delivers tasks
- Task consumer → Session runner: runs session
- Session runner → Agent context: builds context
- Agent context → Memory and knowledge: retrieves memory
- Agent context → LLM providers: requests replies
- Session runner → Speech recognition: transcribes input
- Session runner → Speech synthesis: synthesizes speech
- Session runner → Speech rendering: renders speech
- Speech rendering → Avatar model adapters: infers frames
- Avatar model adapters → Avatar and voice assets: uses avatar assets
- Speech rendering → Media fan-out: emits media
- Media fan-out → WebRTC playback: delivers playback
- WebRTC playback → Conversation WebUI: plays streams
- Media fan-out → Live outputs: fans out media
- REST API → Video creation: exposes creation
- REST API → Video cloning: exposes cloning
- REST API → Persona packages: manages personas
- REST API → Memory and knowledge: manages memory