How opensqz/minicpm-v-cookbook works
1. This repository is a cookbook, not a single application. Its architecture is organized around “recipes” for the MiniCPM-V / MiniCPM-o model family: inference examples, fine-tuning pipelines, quantization workflows, deployment guides, and runnable demos.
2. The documentation system is the top-level integration layer. docs/ contains the Sphinx site, with localized content under docs/locales/zh_CN, and the root README.md serves as a navigation index that points to all major subsystems and supported runtimes.
3. The inference subsystem lives in inference/. It is a set of standalone examples for core multimodal tasks: single-image QA, multi-image QA, video understanding, PDF/document parsing, OCR, grounding, speech-to-text, text-to-speech, and voice cloning. Sample media in inference/assets/ is part of the data path for reproducible demos.
4. The fine-tuning subsystem is in finetune/. It includes dataset guidance, training scripts (finetune.py, trainer.py), DeepSpeed configs, and framework-specific notes for Transformers, LLaMA-Factory, SWIFT, and Align-anything. This layer connects prepared datasets to model adaptation workflows.
5. The quantization subsystem is in quantization/. It provides AWQ, BNB, and GGUF conversion scripts plus docs for producing smaller, deployable model artifacts. These outputs are intended to feed directly into the deployment recipes rather than a separate serving stack.
6. The deployment subsystem is split by serving runtime in deployment/: vLLM and SGLang for high-throughput GPU serving, llama.cpp for CPU/on-device use, and Ollama for simplified local deployment. These are the canonical backends referenced by the cookbook’s serving matrix.
7. The demo subsystem is where the repo becomes interactive. demo/ contains multiple frontends and backend variants: a FastAPI-style web demo, an iOS demo, a Gradio demo, an OpenWebUI integration, and an “omni_stream” real-time web client/server pair. Each demo is a different UI/runtime packaging of the same model capabilities.
8. The most complex runtime architecture is in demo/web_demo/WebRTC_Demo/. It combines a Vue/Vite frontend (o45-frontend), a Python HTTP backend (cpp_server and omini_backend_code), nginx, and docker-compose. This stack is designed for real-time voice/video interaction, with LiveKit used for room/session coordination and WebRTC-style streaming.
9. Inside that web demo backend, the main boundaries are clear: request/API handling (api/, voice_chat/routes.py), session/token state (voice_chat/entity/), media/audio processing (common/utils/, voice_chat/vad/), and service coordination (services/, including heartbeat and inference service management). Redis is used for shared state, and ONNX VAD models are bundled for speech activity detection.
10. The frontend architecture is component-heavy and mode-driven. The Vue app under o45-frontend/src/ is split into reusable components, hooks, device-aware routers, and view variants for mobile/tablet/web. It manages voice/video call states, model configuration, timbre selection, playback, responsiveness, and network diagnostics, with multiple versioned components showing an evolving real-time UX.
11. The demo/web_demo/omni_stream/ area is a second real-time streaming stack with the same pattern: a Vue web client, a Python model server, VAD utilities, and bundled reference audio. It appears optimized for omni-modal streaming interactions, separate from the WebRTC demo but sharing the same model-serving philosophy.
12. Shared assets and configuration form the repository’s support layer: reference WAV files, example images/videos, ONNX VAD weights, nginx configs, docker-compose files, and environment files. These are what make the cookbook reproducible across local, containerized, and device-specific deployments.
13. Overall data flow is: user-facing demo or example → frontend/UI or script → API/service layer → model backend or external serving runtime → generated text/audio/video output. The repo’s main architectural concern is not a single codebase’s internal business logic, but the interoperability between model formats, serving engines, and multimodal interaction frontends.
Main entry points: deployment/README.md Deploy · README.md README · demo/README.md Demo · WebRTC_Demo/README.md WebRTC demo
Components 34
Docs
Top-level documentation and localized site content.
- README · Repository entry point and cookbook navigator.
README.md - Sphinx site · Primary documentation system and localized content.
docs/source/index.rst - Inference docs · Example pages for multimodal inference tasks.
docs/source/inference/transformers.md - Fine-tune docs · Training and adaptation guidance for supported frameworks.
docs/source/finetune/fintune.md - Quant docs · Documentation for quantization workflows and formats.
docs/source/quantization/awq.md - Deploy docs · Guides for runtime-specific local serving stacks.
docs/source/run_locally/llama.cpp.md
Inference
Standalone multimodal usage recipes and sample media.
- Inference · Standalone multimodal inference recipes and assets.
inference/README.md - Image QA · Single-image question answering and reasoning.
inference/single_image.md - Multimodal tasks · Multi-image, video, OCR, PDF parsing, grounding, and scene text recognition.
inference/multi_images.md - Speech tasks · Speech-to-text, text-to-speech, and voice cloning examples.
inference/speech2text.md - Sample media · Reference audio/video/pdf files used by inference recipes.
inference/assets
Fine-tune
Training pipelines, dataset guidance, and framework-specific notes.
- Fine-tune · Model adaptation workflows and training entry points.
finetune/readme.md - Datasets · Dataset guidance and preparation notes for training.
finetune/dataset_guidance.md - Trainer · Main fine-tuning scripts and training orchestration.
finetune/finetune.py - Frameworks · Transformers, LLaMA-Factory, SWIFT, and Align-anything guidance.
finetune/finetune_llamafactory.md - DeepSpeed · Distributed training configuration files.
finetune/ds_config_zero2.json
Quantization
Artifact conversion workflows for smaller deployable model variants.
- Quantize · Quantization workflows for deployable artifacts.
quantization/README.md - AWQ · Activation-aware quantization scripts and notes.
quantization/awq/minicpm-v4_5_awq_quantize.py - BNB · Bitsandbytes quantization workflow.
quantization/bnb/minicpm-v4_5_bnb_quantize.py - GGUF · GGUF conversion workflow for local runtimes.
quantization/gguf/minicpm-v4_gguf_quantize.md
Deployment
Serving backends by runtime and target environment.
- Deploy · Canonical serving backends grouped by runtime.
deployment/README.md - vLLM · High-throughput GPU serving recipes.
deployment/vllm/README.md - SGLang · Alternative GPU serving backend and guides.
deployment/sglang/README.md - llama.cpp · CPU and on-device deployment recipes.
deployment/llama.cpp/README.md - Ollama · Simplified local deployment guides.
deployment/ollama/README.md
Demos
Interactive frontends and real-time runtime packaging.
- Demo · Interactive frontends and runtime-specific demo packaging.
demo/README.md - iOS demo · Native iOS packaging and usage notes.
demo/ios_demo/ios.md - Gradio demo · Gradio-based demo client/server variants.
demo/web_demo/gradio/README.md - OpenWebUI · OpenWebUI integration for conversational use.
demo/web_demo/openwebui/README.md - Omni stream · Real-time streaming web client and server pair.
demo/web_demo/omni_stream/omni_web_client/README.md - WebRTC demo · Full voice/video interaction stack with frontend, backend, nginx, and compose.
demo/web_demo/WebRTC_Demo/README.md - Frontend · Device-aware Vue/Vite UI with reusable components and route variants.
demo/web_demo/WebRTC_Demo/o45-frontend/src/App.vue - Backend · Python backend for API routes, session state, media processing, and service coordination.
demo/web_demo/WebRTC_Demo/omini_backend_code/code/main.py - CPP server · Standalone HTTP server used in the WebRTC demo packaging.
demo/web_demo/WebRTC_Demo/cpp_server/minicpmo_cpp_http_server.py
Connections 41
- README → Sphinx site: introduces · README points into the docs site and cookbook structure.
- Sphinx site → Inference docs: indexes · Documentation site includes inference recipes.
- Sphinx site → Fine-tune docs: indexes · Documentation site includes fine-tuning recipes.
- Sphinx site → Quant docs: indexes · Documentation site includes quantization recipes.
- Sphinx site → Deploy docs: indexes · Documentation site includes local deployment guides.
- README → Inference: links · Root README navigates to inference examples.
- README → Fine-tune: links · Root README navigates to fine-tuning workflows.
- README → Quantize: links · Root README navigates to quantization workflows.
- README → Deploy: links · Root README navigates to deployment backends.
- README → Demo: links · Root README navigates to demo packages.
- Inference → Image QA: covers · Inference suite includes single-image QA.
- Inference → Multimodal tasks: covers · Inference suite groups multi-image, video, OCR, PDF, and grounding tasks.
- Inference → Speech tasks: covers · Inference suite includes speech and voice recipes.
- Inference → Sample media: uses · Examples depend on bundled reference media.
- Fine-tune → Datasets: starts with · Training workflows begin with dataset preparation.
- Fine-tune → Trainer: executes · Fine-tuning is driven by training scripts.
- Fine-tune → Frameworks: adapts to · Fine-tuning supports multiple training frameworks.
- Fine-tune → DeepSpeed: configures · DeepSpeed configs support distributed training.
- Quantize → AWQ: includes · Quantization suite includes AWQ conversion.
- Quantize → BNB: includes · Quantization suite includes BNB conversion.
- Quantize → GGUF: includes · Quantization suite includes GGUF conversion.
- Quantize → Deploy: feeds · Quantized artifacts are intended for deployment backends.
- Deploy → vLLM: routes to · Deployment matrix includes vLLM serving.
- Deploy → SGLang: routes to · Deployment matrix includes SGLang serving.
- Deploy → llama.cpp: routes to · Deployment matrix includes llama.cpp serving.
- Deploy → Ollama: routes to · Deployment matrix includes Ollama serving.
- Demo → iOS demo: includes · Demo suite includes iOS packaging.
- Demo → Gradio demo: includes · Demo suite includes Gradio frontends.
- Demo → OpenWebUI: includes · Demo suite includes OpenWebUI integration.
- Demo → Omni stream: includes · Demo suite includes a real-time omni stream stack.
- Demo → WebRTC demo: includes · Demo suite includes the WebRTC real-time stack.
- WebRTC demo → Frontend: hosts · WebRTC demo bundles a Vue/Vite frontend.
- WebRTC demo → Backend: hosts · WebRTC demo bundles the Python backend.
- WebRTC demo → CPP server: hosts · WebRTC demo also includes a standalone HTTP server variant.
- Frontend → Backend: calls · UI communicates with backend APIs and session services.
- Frontend → WebRTC demo: served by · Frontend is packaged inside the real-time demo stack.
- Backend → CPP server: coordinates · Backend and HTTP server are alternative or cooperating runtime pieces.
- Backend → Deploy: uses · Backend delegates model execution to serving runtimes.
- Omni stream → Frontend: pairs with · Omni stream uses a web client frontend.
- Omni stream → Backend: pairs with · Omni stream has a Python model-serving backend.
- WebRTC demo → Sample media: uses · Real-time demo relies on bundled audio/media assets.