How openbmb/minicpm-o works
This project is a full‐scale, multimodal large language model (MLLM) initiative—MiniCPM‐o/MiniCPM‐V—that integrates vision, speech, text, and video processing into an end‑to‑end system. Its aim is to deliver GPT‑4o–level performance (or higher) in tasks such as live streaming, multimodal chatting, and real‑time inference on devices like phones and iPads. The repository contains components for model inference, evaluation, fine‑tuning, quantization, and web demo integrations.
Core Model Pipeline (Backbone)
• At the center is the end‑to‑end multimodal backbone where the various modality encoders (for images, video frames, audio) and decoders (TTS, text generators) are “glued” together. This central engine implements the omni‑modal streaming mechanism (via time-division multiplexing, TDM) that turns parallel data streams (visual, audio) into sequential tokens.
• Submodules for:
– Modality-specific preprocessing: Image decoding (from photos, video frames), audio decoding (using libraries like librosa) and video-to-frame extraction.
– The LLM core: the primary transformer network that processes tokens from different modalities end‑to‑end.
– Post‑processing and output decoding: text generation, TTS synthesis, and audio reconstruction.
Data Input and Preprocessing Modules
• Separate blocks for handling file-based as well as streaming inputs:
– Image inputs, video inputs (with frame extraction and uniform sampling, as seen in the “get_video_chunk_content” function), and audio streams.
• Each of these is preprocessed before feeding into the model core (e.g., using PIL for images, decord for video, librosa for audio).
Inference & Streaming Mechanism
• Distinguish between regular “chat” inference (batch-style) and streaming inference.
• A “prefill” block that resets sessions and feeds sequential macro-units (combining image, audio, text) into the model.
• That the system supports both offline inference as well as a live streaming inference (multimodal live streaming) with TDM.
Fine-Tuning and Evaluation Modules
• Fine-tuning: Represent the “finetune” directory which contains scripts, configuration files (JSON and shell scripts) and trainer code to adjust model parameters.
• Evaluation: Include the “eval_mm” and “vqaeval” directories dedicated to benchmarking on various datasets (e.g., OCRBench, StreamingBench, multi-turn conversation, etc.). These modules are responsible for data loading, running multiple benchmarks, and aggregating results.
• Configuration and documentation files (under “docs”) provide technical reports, best practices, and FAQs that also guide deployment.
Deployment, Quantization, and External Integrations
• Deployment: A separate block represents web demos, model servers (chatbot_web_demo, model_server in the “web_demos” directory), and integration with node-based web frontends (with Dockerfiles, nginx config, and build scripts).
• Quantization and Optimizations: The “quantize” folder contains scripts (e.g., for bnb_quantize.py) indicating optional model size reduction and improved inference efficiency.
• External Inference Frameworks: Show links with forks/integration for llama.cpp, ollama, and vLLM which are used for efficient CPU and multi-GPU inference.
Client and External Services
• Frontends: The system supplies a mobile/web demo interface (evidenced by “web_demos”) that send user queries (text, image, video, audio) via a REST/HTTP API.
• External model hubs: There are direct links to Hugging Face, ModelBest, and deployment references in the README.