# openbmb/minicpm-o architecture

> Architecture diagram of the GitHub repository openbmb/minicpm-o, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/minicpm-o
- Repository: https://github.com/openbmb/minicpm-o

## Overview

This project is a full‐scale, multimodal large language model (MLLM) initiative—MiniCPM‐o/MiniCPM‐V—that integrates vision, speech, text, and video processing into an end‑to‑end system. Its aim is to deliver GPT‑4o–level performance (or higher) in tasks such as live streaming, multimodal chatting, and real‑time inference on devices like phones and iPads. The repository contains components for model inference, evaluation, fine‑tuning, quantization, and web demo integrations.

##### Core Model Pipeline (Backbone)
- At the center is the end‑to‑end multimodal backbone where the various modality encoders (for images, video frames, audio) and decoders (TTS, text generators) are “glued” together. This central engine implements the omni‑modal streaming mechanism (via time-division multiplexing, TDM) that turns parallel data streams (visual, audio) into sequential tokens.
- Submodules for:
– Modality-specific preprocessing: Image decoding (from photos, video frames), audio decoding (using libraries like librosa) and video-to-frame extraction.
– The LLM core: the primary transformer network that processes tokens from different modalities end‑to‑end.
– Post‑processing and output decoding: text generation, TTS synthesis, and audio reconstruction.

##### Data Input and Preprocessing Modules
- Separate blocks for handling file-based as well as streaming inputs:
– Image inputs, video inputs (with frame extraction and uniform sampling, as seen in the “get_video_chunk_content” function), and audio streams.
- Each of these is preprocessed before feeding into the model core (e.g., using PIL for images, decord for video, librosa for audio).

##### Inference & Streaming Mechanism
- Distinguish between regular “chat” inference (batch-style) and streaming inference.
- A “prefill” block that resets sessions and feeds sequential macro-units (combining image, audio, text) into the model.
- That the system supports both offline inference as well as a live streaming inference (multimodal live streaming) with TDM.

##### Fine-Tuning and Evaluation Modules
- Fine-tuning: Represent the “finetune” directory which contains scripts, configuration files (JSON and shell scripts) and trainer code to adjust model parameters.
- Evaluation: Include the “eval_mm” and “vqaeval” directories dedicated to benchmarking on various datasets (e.g., OCRBench, StreamingBench, multi-turn conversation, etc.). These modules are responsible for data loading, running multiple benchmarks, and aggregating results.
- Configuration and documentation files (under “docs”) provide technical reports, best practices, and FAQs that also guide deployment.

##### Deployment, Quantization, and External Integrations
- Deployment: A separate block represents web demos, model servers (chatbot_web_demo, model_server in the “web_demos” directory), and integration with node-based web frontends (with Dockerfiles, nginx config, and build scripts).
- Quantization and Optimizations: The “quantize” folder contains scripts (e.g., for bnb_quantize.py) indicating optional model size reduction and improved inference efficiency.
- External Inference Frameworks: Show links with forks/integration for llama.cpp, ollama, and vLLM which are used for efficient CPU and multi-GPU inference.

##### Client and External Services
- Frontends: The system supplies a mobile/web demo interface (evidenced by “web_demos”) that send user queries (text, image, video, audio) via a REST/HTTP API.
- External model hubs: There are direct links to Hugging Face, ModelBest, and deployment references in the README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    INPUT["Input Preprocessing\n(File & Streaming Inputs)"]:::ingestion
    CORE["Core Model Pipeline (Backbone)\n[Modality Preprocessing | LLM Core | Postprocessing]"]:::core
    INFER["Inference & Streaming Mechanism\n(Prefill & TDM)"]:::inference
    FINETUNE["Fine-Tuning Module"]:::eval
    EVALUATION["Evaluation Modules"]:::eval
    QUANT["Quantization & Optimization"]:::deploy
    DEPLOY["Deployment & External Integrations\n(Web Demo Frontend)"]:::deploy
    CLIENT["Client & External Services\n(Mobile/Web Demo)"]:::external

    %% Main Flow
    INPUT -->|"preprocessed"| CORE
    CORE -->|"stream tokens"| INFER
    CORE -->|"model updates"| FINETUNE
    CORE -->|"benchmarking"| EVALUATION
    CORE -->|"optimize model"| QUANT
    QUANT -->|"build deployment"| DEPLOY
    DEPLOY -->|"serve responses"| CLIENT

    %% Feedback loop for multi-turn conversation
    CLIENT -->|"user feedback"| INFER
    INFER -->|"iterative response"| CORE

    %% Click Events
    click INPUT "https://github.com/openbmb/minicpm-o/blob/main/finetune/dataset.py"
    click CORE "https://github.com/openbmb/minicpm-o/blob/main/omnilmm/model/omnilmm.py"
    click INFER "https://github.com/openbmb/minicpm-o/blob/main/omnilmm/conversation.py"
    click FINETUNE "https://github.com/openbmb/minicpm-o/tree/main/finetune/"
    click EVALUATION "https://github.com/openbmb/minicpm-o/tree/main/eval_mm/"
    click QUANT "https://github.com/openbmb/minicpm-o/blob/main/quantize/bnb_quantize.py"
    click DEPLOY "https://github.com/openbmb/minicpm-o/tree/main/web_demos/"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef core fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef inference fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef eval fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef deploy fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef external fill:#e2e3e5,stroke:#41464b,stroke-width:2px;
```
