# openbmb/minicpm-v architecture

> Architecture diagram of the GitHub repository openbmb/minicpm-v, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/minicpm-v
- Repository: https://github.com/openbmb/minicpm-v

## Overview

MiniCPM-V and MiniCPM-o are multimodal language models intended for efficient deployment: V handles image, video, and text understanding, while o adds streaming audio/video interaction and speech output. The interactive workflow runs from a user-facing demo through a server to model inference; separate branches support fine-tuning and benchmark evaluation. The sampled server and inference code support those flows, but model-loading details and several README-described capabilities were not inspected, so links to their internals are omitted where uncertain.

Main entry points: [`omnilmm/model/omnilmm.py`](https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/omnilmm.py) (Omni model), [`eval_mm/vlmevalkit/vlmeval/inference.py`](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference.py) (Image inference), [`eval_mm/vlmevalkit/vlmeval/inference_video.py`](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference_video.py) (Video inference), [`eval_mm/vlmevalkit/vlmeval/inference_mt.py`](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference_mt.py) (Multi-turn inference)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interaction["Interactive demos"]
  node_web_ui["Web interface<br/>[VideoCall.vue]"]
  node_web_api["HTTP client<br/>[useHttp.js]"]
  node_voice_capture["Audio encoding<br/>[useVoice.js]"]
  node_model_server["Omni server<br/>[model_server.py]"]
  node_demo_26["V2.6 demo<br/>[web_demo_2.6.py]"]
end

subgraph group_model["Model runtime"]
  node_chat_entry["Chat entry<br/>[chat.py]"]
  node_omnilmm["Omni model<br/>[omnilmm.py]"]
  node_resampler["Vision resampler<br/>[resampler.py]"]
  node_model_utils["Model utilities<br/>[utils.py]"]
end

subgraph group_training["Fine-tuning"]
  node_finetune_data["Training data<br/>[dataset.py]"]
  node_trainer["Trainer<br/>[trainer.py]"]
  node_finetune_run["Fine-tuning<br/>[finetune.py]"]
end

subgraph group_evaluation["Benchmark evaluation"]
  node_image_datasets["Image datasets<br/>[image_base.py]"]
  node_video_datasets["Video datasets<br/>[video_base.py]"]
  node_vlm_inference["Image inference<br/>[inference.py]"]
  node_video_inference["Video inference<br/>[inference_video.py]"]
  node_mt_inference["Multi-turn inference<br/>[inference_mt.py]"]
  node_model_registry["Model registry<br/>[config.py]"]
  node_results["Evaluation results<br/>[result_transfer.py]"]
  node_vqa_eval["VQA evaluation<br/>[eval.py]"]
  node_vqa_data["VQA dataset<br/>[vqa_dataset.py]"]
end

node_user(("User"))
node_eval_user(("Evaluator"))

node_user -->|"interacts"| node_web_ui
node_web_ui -->|"encodes audio"| node_voice_capture
node_web_ui -->|"sends requests"| node_web_api
node_web_api -->|"HTTP requests"| node_model_server
node_model_server -->|"handles streams"| node_model_server
node_user -->|"submits media"| node_demo_26
node_demo_26 -->|"invokes chat"| node_chat_entry
node_chat_entry -->|"runs model"| node_omnilmm
node_omnilmm -->|"uses vision encoding"| node_resampler
node_omnilmm -->|"uses utilities"| node_model_utils
node_finetune_run -->|"loads examples"| node_finetune_data
node_finetune_run -->|"starts training"| node_trainer
node_trainer -->|"trains model"| node_omnilmm
node_eval_user -->|"selects benchmark"| node_image_datasets
node_eval_user -->|"selects benchmark"| node_video_datasets
node_vlm_inference -->|"builds prompts"| node_image_datasets
node_video_inference -->|"builds prompts"| node_video_datasets
node_mt_inference -->|"builds prompts"| node_image_datasets
node_vlm_inference -->|"resolves model"| node_model_registry
node_video_inference -->|"resolves model"| node_model_registry
node_mt_inference -->|"resolves model"| node_model_registry
node_vlm_inference -->|"writes predictions"| node_results
node_video_inference -->|"writes predictions"| node_results
node_mt_inference -->|"writes predictions"| node_results
node_vqa_eval -->|"loads samples"| node_vqa_data

click node_web_ui "https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/web_server/src/views/home/components/VideoCall.vue"
click node_web_api "https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/web_server/src/hooks/useHttp.js"
click node_voice_capture "https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/web_server/src/hooks/useVoice.js"
click node_model_server "https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/model_server.py"
click node_demo_26 "https://github.com/openbmb/minicpm-v/blob/main/web_demos/web_demo_2.6.py"
click node_chat_entry "https://github.com/openbmb/minicpm-v/blob/main/chat.py"
click node_omnilmm "https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/omnilmm.py"
click node_resampler "https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/resampler.py"
click node_model_utils "https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/utils.py"
click node_finetune_data "https://github.com/openbmb/minicpm-v/blob/main/finetune/dataset.py"
click node_trainer "https://github.com/openbmb/minicpm-v/blob/main/finetune/trainer.py"
click node_finetune_run "https://github.com/openbmb/minicpm-v/blob/main/finetune/finetune.py"
click node_image_datasets "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/dataset/image_base.py"
click node_video_datasets "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/dataset/video_base.py"
click node_vlm_inference "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference.py"
click node_video_inference "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference_video.py"
click node_mt_inference "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference_mt.py"
click node_model_registry "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/config.py"
click node_results "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/utils/result_transfer.py"
click node_vqa_eval "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vqaeval/eval.py"
click node_vqa_data "https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vqaeval/datasets/vqa_dataset.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web_ui,node_web_api,node_voice_capture,node_model_server,node_demo_26,node_user toneBlue
class node_chat_entry,node_omnilmm,node_resampler,node_model_utils toneAmber
class node_finetune_data,node_trainer,node_finetune_run toneMint
class node_image_datasets,node_video_datasets,node_vlm_inference,node_video_inference,node_mt_inference,node_model_registry,node_results,node_vqa_eval,node_vqa_data toneRose
class node_eval_user toneIndigo
```

## Components

### Interactive demos

- [Web interface](https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/web_server/src/views/home/components/VideoCall.vue): `web_demos/minicpm-o_2.6/web_server/src/views/home/components/VideoCall.vue`
- [HTTP client](https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/web_server/src/hooks/useHttp.js): `web_demos/minicpm-o_2.6/web_server/src/hooks/useHttp.js`
- [Audio encoding](https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/web_server/src/hooks/useVoice.js): `web_demos/minicpm-o_2.6/web_server/src/hooks/useVoice.js`
- [Omni server](https://github.com/openbmb/minicpm-v/blob/main/web_demos/minicpm-o_2.6/model_server.py): `web_demos/minicpm-o_2.6/model_server.py`
- [V2.6 demo](https://github.com/openbmb/minicpm-v/blob/main/web_demos/web_demo_2.6.py): `web_demos/web_demo_2.6.py`

### Model runtime

- [Chat entry](https://github.com/openbmb/minicpm-v/blob/main/chat.py): `chat.py`
- [Omni model](https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/omnilmm.py): `omnilmm/model/omnilmm.py`
- [Vision resampler](https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/resampler.py): `omnilmm/model/resampler.py`
- [Model utilities](https://github.com/openbmb/minicpm-v/blob/main/omnilmm/model/utils.py): `omnilmm/model/utils.py`

### Fine-tuning

- [Training data](https://github.com/openbmb/minicpm-v/blob/main/finetune/dataset.py): `finetune/dataset.py`
- [Trainer](https://github.com/openbmb/minicpm-v/blob/main/finetune/trainer.py): `finetune/trainer.py`
- [Fine-tuning](https://github.com/openbmb/minicpm-v/blob/main/finetune/finetune.py): `finetune/finetune.py`

### Benchmark evaluation

- [Image datasets](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/dataset/image_base.py): `eval_mm/vlmevalkit/vlmeval/dataset/image_base.py`
- [Video datasets](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/dataset/video_base.py): `eval_mm/vlmevalkit/vlmeval/dataset/video_base.py`
- [Image inference](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference.py): `eval_mm/vlmevalkit/vlmeval/inference.py`
- [Video inference](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference_video.py): `eval_mm/vlmevalkit/vlmeval/inference_video.py`
- [Multi-turn inference](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/inference_mt.py): `eval_mm/vlmevalkit/vlmeval/inference_mt.py`
- [Model registry](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/config.py): `eval_mm/vlmevalkit/vlmeval/config.py`
- [Evaluation results](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vlmevalkit/vlmeval/utils/result_transfer.py): `eval_mm/vlmevalkit/vlmeval/utils/result_transfer.py`
- [VQA evaluation](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vqaeval/eval.py): `eval_mm/vqaeval/eval.py`
- [VQA dataset](https://github.com/openbmb/minicpm-v/blob/main/eval_mm/vqaeval/datasets/vqa_dataset.py): `eval_mm/vqaeval/datasets/vqa_dataset.py`

### Other components

- **User**
- **Evaluator**

## Connections

- User → Web interface: interacts
- Web interface → Audio encoding: encodes audio
- Web interface → HTTP client: sends requests
- HTTP client → Omni server: HTTP requests
- Omni server → Omni server: handles streams
- User → V2.6 demo: submits media
- V2.6 demo → Chat entry: invokes chat
- Chat entry → Omni model: runs model
- Omni model → Vision resampler: uses vision encoding
- Omni model → Model utilities: uses utilities
- Fine-tuning → Training data: loads examples
- Fine-tuning → Trainer: starts training
- Trainer → Omni model: trains model
- Evaluator → Image datasets: selects benchmark
- Evaluator → Video datasets: selects benchmark
- Image inference → Image datasets: builds prompts
- Video inference → Video datasets: builds prompts
- Multi-turn inference → Image datasets: builds prompts
- Image inference → Model registry: resolves model
- Video inference → Model registry: resolves model
- Multi-turn inference → Model registry: resolves model
- Image inference → Evaluation results: writes predictions
- Video inference → Evaluation results: writes predictions
- Multi-turn inference → Evaluation results: writes predictions
- VQA evaluation → VQA dataset: loads samples