How openbmb/minicpm-v works
MiniCPM-V and MiniCPM-o are multimodal language models intended for efficient deployment: V handles image, video, and text understanding, while o adds streaming audio/video interaction and speech output. The interactive workflow runs from a user-facing demo through a server to model inference; separate branches support fine-tuning and benchmark evaluation. The sampled server and inference code support those flows, but model-loading details and several README-described capabilities were not inspected, so links to their internals are omitted where uncertain.
Main entry points: omnilmm.py Omni model · inference.py Image inference · inference_video.py Video inference · inference_mt.py Multi-turn inference
Components 23
Interactive demos
- Web interface
web_demos/minicpm-o_2.6/web_server/src/views/home/components/VideoCall.vue - HTTP client
web_demos/minicpm-o_2.6/web_server/src/hooks/useHttp.js - Audio encoding
web_demos/minicpm-o_2.6/web_server/src/hooks/useVoice.js - Omni server
web_demos/minicpm-o_2.6/model_server.py - V2.6 demo
web_demos/web_demo_2.6.py
Model runtime
- Chat entry
chat.py - Omni model
omnilmm/model/omnilmm.py - Vision resampler
omnilmm/model/resampler.py - Model utilities
omnilmm/model/utils.py
Fine-tuning
- Training data
finetune/dataset.py - Trainer
finetune/trainer.py - Fine-tuning
finetune/finetune.py
Benchmark evaluation
- Image datasets
eval_mm/vlmevalkit/vlmeval/dataset/image_base.py - Video datasets
eval_mm/vlmevalkit/vlmeval/dataset/video_base.py - Image inference
eval_mm/vlmevalkit/vlmeval/inference.py - Video inference
eval_mm/vlmevalkit/vlmeval/inference_video.py - Multi-turn inference
eval_mm/vlmevalkit/vlmeval/inference_mt.py - Model registry
eval_mm/vlmevalkit/vlmeval/config.py - Evaluation results
eval_mm/vlmevalkit/vlmeval/utils/result_transfer.py - VQA evaluation
eval_mm/vqaeval/eval.py - VQA dataset
eval_mm/vqaeval/datasets/vqa_dataset.py
Other components
- User
- Evaluator
Connections 25
- User → Web interface: interacts
- Web interface → Audio encoding: encodes audio
- Web interface → HTTP client: sends requests
- HTTP client → Omni server: HTTP requests
- Omni server → Omni server: handles streams
- User → V2.6 demo: submits media
- V2.6 demo → Chat entry: invokes chat
- Chat entry → Omni model: runs model
- Omni model → Vision resampler: uses vision encoding
- Omni model → Model utilities: uses utilities
- Fine-tuning → Training data: loads examples
- Fine-tuning → Trainer: starts training
- Trainer → Omni model: trains model
- Evaluator → Image datasets: selects benchmark
- Evaluator → Video datasets: selects benchmark
- Image inference → Image datasets: builds prompts
- Video inference → Video datasets: builds prompts
- Multi-turn inference → Image datasets: builds prompts
- Image inference → Model registry: resolves model
- Video inference → Model registry: resolves model
- Multi-turn inference → Model registry: resolves model
- Image inference → Evaluation results: writes predictions
- Video inference → Evaluation results: writes predictions
- Multi-turn inference → Evaluation results: writes predictions
- VQA evaluation → VQA dataset: loads samples