# openbmb/minicpm-o-demo architecture

> Architecture diagram of the GitHub repository openbmb/minicpm-o-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/minicpm-o-demo
- Repository: https://github.com/openbmb/minicpm-o-demo

## Overview

This repository is a MiniCPM-o 4.5 demo service for turn-based multimodal chat and audio/video full-duplex conversations. The main path is browser interaction through the Gateway and a GPU Worker to a runtime session, backend inference, and streamed results. The map also shows the model-processing layer and the Gateway’s documented queue/session-recording role. README statements establish the service topology; SOURCE INDEX references and excerpts support several internal links. Unsampled implementations and their finer-grained connections are not inferred.

Main entry points: [`static/omni/omni-app.js`](https://github.com/openbmb/minicpm-o-demo/blob/main/static/omni/omni-app.js) (Omni duplex UI), [`worker.py`](https://github.com/openbmb/minicpm-o-demo/blob/main/worker.py) (GPU worker), [`gateway.py`](https://github.com/openbmb/minicpm-o-demo/blob/main/gateway.py) (Gateway), [`core/processors/unified.py`](https://github.com/openbmb/minicpm-o-demo/blob/main/core/processors/unified.py) (Unified processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["User interfaces"]
  node_mobile["Mobile app<br/>[App.tsx]"]
  node_duplex_ui["Omni duplex UI<br/>[omni-app.js]"]
  node_mixer["Audio mixer"]
  node_video_recorder["Video recording"]
end

subgraph group_service["Request service"]
  node_gateway["Gateway<br/>[gateway.py]"]
  node_worker_pool["Worker pool<br/>[worker_pool.py]"]
  node_recording["Session recording"]
end

subgraph group_runtime["Inference runtime"]
  node_worker["GPU worker<br/>[worker.py]"]
  node_worker_state["Worker state<br/>[worker_state.py]"]
  node_runtime_session["Runtime session<br/>[session.py]"]
  node_backend_client["Backend client<br/>[backend_client.py]"]
  node_backend_server["Backend server<br/>[server.py]"]
end

subgraph group_model["Model processing"]
  node_backend_factory["Backend factory<br/>[backend_factory.py]"]
  node_capabilities["Mode capabilities<br/>[capabilities.py]"]
  node_processor["Unified processor<br/>[unified.py]"]
  node_model_runtime["Omni model"]
  node_vision_encoder["Vision encoder"]
  node_media_processing["Media processing"]
end

node_user(("User"))

node_user -->|"uses"| node_mobile
node_user -->|"uses"| node_duplex_ui
node_mobile -->|"sends requests"| node_gateway
node_duplex_ui -->|"sends requests"| node_gateway
node_duplex_ui -->|"uses"| node_mixer
node_duplex_ui -->|"uses"| node_video_recorder
node_gateway -->|"routes and queues"| node_worker_pool
node_gateway -->|"records sessions"| node_recording
node_worker_pool -->|"dispatches requests"| node_worker
node_worker -->|"tracks state"| node_worker_state
node_worker -->|"opens sessions"| node_runtime_session
node_runtime_session -->|"uses"| node_backend_client
node_worker -->|"forwards protocol"| node_backend_server
node_backend_server -->|"creates backend"| node_backend_factory
node_backend_factory -->|"selects backend"| node_processor
node_processor -->|"uses mode definitions"| node_capabilities
node_processor -->|"runs inference"| node_model_runtime
node_model_runtime -->|"uses"| node_vision_encoder
node_model_runtime -->|"uses"| node_media_processing

click node_mobile "https://github.com/openbmb/minicpm-o-demo/blob/main/frontend/mobile/src/App.tsx"
click node_duplex_ui "https://github.com/openbmb/minicpm-o-demo/blob/main/static/omni/omni-app.js"
click node_mixer "https://github.com/openbmb/minicpm-o-demo/blob/main/static/duplex/lib/mixer-controller.js"
click node_video_recorder "https://github.com/openbmb/minicpm-o-demo/blob/main/static/duplex/lib/session-video-recorder.js"
click node_gateway "https://github.com/openbmb/minicpm-o-demo/blob/main/gateway.py"
click node_worker_pool "https://github.com/openbmb/minicpm-o-demo/blob/main/gateway_modules/worker_pool.py"
click node_recording "https://github.com/openbmb/minicpm-o-demo/blob/main/gateway_modules/session_recording.py"
click node_worker "https://github.com/openbmb/minicpm-o-demo/blob/main/worker.py"
click node_worker_state "https://github.com/openbmb/minicpm-o-demo/blob/main/worker_state.py"
click node_runtime_session "https://github.com/openbmb/minicpm-o-demo/blob/main/runtime/session.py"
click node_backend_client "https://github.com/openbmb/minicpm-o-demo/blob/main/runtime/backend_client.py"
click node_backend_server "https://github.com/openbmb/minicpm-o-demo/blob/main/py_backend/server.py"
click node_backend_factory "https://github.com/openbmb/minicpm-o-demo/blob/main/core/processors/backend_factory.py"
click node_capabilities "https://github.com/openbmb/minicpm-o-demo/blob/main/core/capabilities.py"
click node_processor "https://github.com/openbmb/minicpm-o-demo/blob/main/core/processors/unified.py"
click node_model_runtime "https://github.com/openbmb/minicpm-o-demo/blob/main/MiniCPMO45/modeling_minicpmo_unified.py"
click node_vision_encoder "https://github.com/openbmb/minicpm-o-demo/blob/main/MiniCPMO45/modeling_navit_siglip.py"
click node_media_processing "https://github.com/openbmb/minicpm-o-demo/blob/main/MiniCPMO45/processing_minicpmo.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mobile,node_duplex_ui,node_mixer,node_video_recorder,node_user toneBlue
class node_gateway,node_worker_pool,node_recording toneAmber
class node_worker,node_worker_state,node_runtime_session,node_backend_client,node_backend_server toneMint
class node_backend_factory,node_capabilities,node_processor,node_model_runtime,node_vision_encoder,node_media_processing toneRose
```

## Components

### User interfaces

- [Mobile app](https://github.com/openbmb/minicpm-o-demo/blob/main/frontend/mobile/src/App.tsx): `frontend/mobile/src/App.tsx`
- [Omni duplex UI](https://github.com/openbmb/minicpm-o-demo/blob/main/static/omni/omni-app.js): `static/omni/omni-app.js`
- [Audio mixer](https://github.com/openbmb/minicpm-o-demo/blob/main/static/duplex/lib/mixer-controller.js): `static/duplex/lib/mixer-controller.js`
- [Video recording](https://github.com/openbmb/minicpm-o-demo/blob/main/static/duplex/lib/session-video-recorder.js): `static/duplex/lib/session-video-recorder.js`

### Request service

- [Gateway](https://github.com/openbmb/minicpm-o-demo/blob/main/gateway.py): `gateway.py`
- [Worker pool](https://github.com/openbmb/minicpm-o-demo/blob/main/gateway_modules/worker_pool.py): `gateway_modules/worker_pool.py`
- [Session recording](https://github.com/openbmb/minicpm-o-demo/blob/main/gateway_modules/session_recording.py): `gateway_modules/session_recording.py`

### Inference runtime

- [GPU worker](https://github.com/openbmb/minicpm-o-demo/blob/main/worker.py): `worker.py`
- [Worker state](https://github.com/openbmb/minicpm-o-demo/blob/main/worker_state.py): `worker_state.py`
- [Runtime session](https://github.com/openbmb/minicpm-o-demo/blob/main/runtime/session.py): `runtime/session.py`
- [Backend client](https://github.com/openbmb/minicpm-o-demo/blob/main/runtime/backend_client.py): `runtime/backend_client.py`
- [Backend server](https://github.com/openbmb/minicpm-o-demo/blob/main/py_backend/server.py): `py_backend/server.py`

### Model processing

- [Backend factory](https://github.com/openbmb/minicpm-o-demo/blob/main/core/processors/backend_factory.py): `core/processors/backend_factory.py`
- [Mode capabilities](https://github.com/openbmb/minicpm-o-demo/blob/main/core/capabilities.py): `core/capabilities.py`
- [Unified processor](https://github.com/openbmb/minicpm-o-demo/blob/main/core/processors/unified.py): `core/processors/unified.py`
- [Omni model](https://github.com/openbmb/minicpm-o-demo/blob/main/MiniCPMO45/modeling_minicpmo_unified.py): `MiniCPMO45/modeling_minicpmo_unified.py`
- [Vision encoder](https://github.com/openbmb/minicpm-o-demo/blob/main/MiniCPMO45/modeling_navit_siglip.py): `MiniCPMO45/modeling_navit_siglip.py`
- [Media processing](https://github.com/openbmb/minicpm-o-demo/blob/main/MiniCPMO45/processing_minicpmo.py): `MiniCPMO45/processing_minicpmo.py`

### Other components

- **User**

## Connections

- User → Mobile app: uses (evidence: `README.md`)
- User → Omni duplex UI: uses (evidence: `README.md`)
- Mobile app → Gateway: sends requests (evidence: `README.md`)
- Omni duplex UI → Gateway: sends requests (evidence: `README.md`)
- Omni duplex UI → Audio mixer: uses (evidence: `static/omni/omni-app.js`)
- Omni duplex UI → Video recording: uses (evidence: `static/omni/omni-app.js`)
- Gateway → Worker pool: routes and queues (evidence: `README.md`)
- Gateway → Session recording: records sessions (evidence: `README.md`)
- Worker pool → GPU worker: dispatches requests (evidence: `README.md`)
- GPU worker → Worker state: tracks state (evidence: `worker.py`)
- GPU worker → Runtime session: opens sessions (evidence: `worker.py`)
- Runtime session → Backend client: uses (evidence: `runtime/session.py`)
- GPU worker → Backend server: forwards protocol (evidence: `README.md`)
- Backend server → Backend factory: creates backend (evidence: `py_backend/server.py`)
- Backend factory → Unified processor: selects backend (evidence: `core/processors/backend_factory.py`)
- Unified processor → Mode capabilities: uses mode definitions (evidence: `core/processors/unified.py`)
- Unified processor → Omni model: runs inference (evidence: `core/processors/unified.py`)
- Omni model → Vision encoder: uses (evidence: `MiniCPMO45/modeling_minicpmo_unified.py`)
- Omni model → Media processing: uses (evidence: `MiniCPMO45/modeling_minicpmo_unified.py`)
