# opendatalab/mineru-vl-utils architecture

> Architecture diagram of the GitHub repository opendatalab/mineru-vl-utils, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendatalab/mineru-vl-utils
- Repository: https://github.com/opendatalab/mineru-vl-utils

## Overview

MinerU VL Utils is a Python client library for document vision-language inference. The principal workflow accepts an image, runs layout detection, prepares region images and prompts, recognizes block content through a selected inference backend, and post-processes the resulting structured blocks. The README documents seven deployment backends; the excerpts show the client’s post-processing and image preparation responsibilities, shared client abstractions, and several backend implementations. Some workflow stages and backend selection details are documented rather than visible in the excerpts, so those links cite the README.

Main entry points: [`mineru_vl_utils/mineru_client.py`](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py) (MinerU client), [`mineru_vl_utils/vlm_client/base_client.py`](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/base_client.py) (VLM client contract), [`mineru_vl_utils/post_process/__init__.py`](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/__init__.py) (Block post-processing), [`mineru_vl_utils/vlm_client/http_client.py`](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/http_client.py) (HTTP backend)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Document workflow"]
  node_mineru["MinerU client<br/>[mineru_client.py]"]
  node_layout["Layout detection<br/>[mineru_client.py]"]
  node_regionprep["Region preparation<br/>[mineru_client.py]"]
  node_blocks["Content blocks<br/>[structs.py]"]
end

subgraph group_backends["Inference backends"]
  node_http["HTTP backend<br/>[http_client.py]"]
  node_transformers["Transformers backend"]
  node_mlx["MLX backend<br/>[mlx_client.py]"]
  node_lmdeploy["LMDeploy backend"]
  node_vllm["vLLM backend"]
  node_asyncvllm["Async vLLM backend"]
  node_llamacpp["llama.cpp backend"]
end

subgraph group_postprocess["Output processing"]
  node_post["Block post-processing<br/>[__init__.py]"]
  node_imageanalysis["Image analysis cleanup"]
  node_tableimage["Table image handling"]
  node_crosspage["Cross-page tables"]
  node_equations["Equation cleanup<br/>[equation_big.py]"]
  node_charts["Chart repetition cleanup"]
  node_mermaid["Mermaid cleanup"]
end

subgraph group_shared["Shared contracts"]
  node_base["VLM client contract<br/>[base_client.py]"]
  node_clientutils["Image utilities<br/>[utils.py]"]
  node_logits["No-repeat sampling"]
end

node_caller(("Application caller"))
node_server["OpenAI-compatible server"]

node_caller -->|"calls"| node_mineru
node_mineru -->|"detects layout"| node_layout
node_mineru -->|"prepares regions"| node_regionprep
node_regionprep -->|"masks table images"| node_tableimage
node_mineru -.->|"selects backend"| node_http
node_mineru -.->|"selects backend"| node_transformers
node_mineru -.->|"selects backend"| node_mlx
node_mineru -.->|"selects backend"| node_lmdeploy
node_mineru -.->|"selects backend"| node_vllm
node_mineru -.->|"selects backend"| node_asyncvllm
node_mineru -.->|"selects backend"| node_llamacpp
node_mineru -->|"processes blocks"| node_post
node_post -->|"updates blocks"| node_blocks
node_post -->|"analyzes images"| node_imageanalysis
node_post -->|"replaces tokens"| node_tableimage
node_post -->|"fixes equations"| node_equations
node_imageanalysis -->|"cleans repetitions"| node_charts
node_imageanalysis -->|"cleans diagrams"| node_mermaid
node_mineru -.->|"handles page tables"| node_crosspage
node_http -->|"implements contract"| node_base
node_transformers -->|"implements contract"| node_base
node_mlx -->|"implements contract"| node_base
node_lmdeploy -->|"implements contract"| node_base
node_vllm -->|"implements contract"| node_base
node_asyncvllm -->|"implements contract"| node_base
node_llamacpp -->|"implements contract"| node_base
node_http -->|"loads images"| node_clientutils
node_vllm -.->|"supports sampling"| node_logits
node_http -->|"sends requests"| node_server
node_mineru -->|"returns results"| node_blocks

click node_mineru "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py"
click node_layout "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py"
click node_regionprep "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py"
click node_blocks "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/structs.py"
click node_http "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/http_client.py"
click node_transformers "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/transformers_client.py"
click node_mlx "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/mlx_client.py"
click node_lmdeploy "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/lmdeploy_engine_client.py"
click node_vllm "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/vllm_engine_client.py"
click node_asyncvllm "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/vllm_async_engine_client.py"
click node_llamacpp "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/llama_cpp_engine_client.py"
click node_base "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/base_client.py"
click node_clientutils "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/utils.py"
click node_post "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/__init__.py"
click node_imageanalysis "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/image_analysis_postprocess.py"
click node_tableimage "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/table_image_processor.py"
click node_crosspage "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/cross_page_table.py"
click node_equations "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/equation_big.py"
click node_charts "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/chart_invalid_repetition.py"
click node_mermaid "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/mermaid_repetition.py"
click node_logits "https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/logits_processor/vllm_v1_no_repeat_ngram.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mineru,node_layout,node_regionprep,node_blocks toneBlue
class node_http,node_transformers,node_mlx,node_lmdeploy,node_vllm,node_asyncvllm,node_llamacpp toneAmber
class node_post,node_imageanalysis,node_tableimage,node_crosspage,node_equations,node_charts,node_mermaid,node_server toneMint
class node_base,node_clientutils,node_logits toneRose
class node_caller toneIndigo
```

## Components

### Document workflow

- [MinerU client](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py): `mineru_vl_utils/mineru_client.py`
- [Layout detection](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py): `mineru_vl_utils/mineru_client.py`
- [Region preparation](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/mineru_client.py): `mineru_vl_utils/mineru_client.py`
- [Content blocks](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/structs.py): `mineru_vl_utils/structs.py`

### Inference backends

- [HTTP backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/http_client.py): `mineru_vl_utils/vlm_client/http_client.py`
- [Transformers backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/transformers_client.py): `mineru_vl_utils/vlm_client/transformers_client.py`
- [MLX backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/mlx_client.py): `mineru_vl_utils/vlm_client/mlx_client.py`
- [LMDeploy backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/lmdeploy_engine_client.py): `mineru_vl_utils/vlm_client/lmdeploy_engine_client.py`
- [vLLM backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/vllm_engine_client.py): `mineru_vl_utils/vlm_client/vllm_engine_client.py`
- [Async vLLM backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/vllm_async_engine_client.py): `mineru_vl_utils/vlm_client/vllm_async_engine_client.py`
- [llama.cpp backend](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/llama_cpp_engine_client.py): `mineru_vl_utils/vlm_client/llama_cpp_engine_client.py`

### Output processing

- [Block post-processing](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/__init__.py): `mineru_vl_utils/post_process/__init__.py`
- [Image analysis cleanup](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/image_analysis_postprocess.py): `mineru_vl_utils/post_process/image_analysis_postprocess.py`
- [Table image handling](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/table_image_processor.py): `mineru_vl_utils/post_process/table_image_processor.py`
- [Cross-page tables](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/cross_page_table.py): `mineru_vl_utils/post_process/cross_page_table.py`
- [Equation cleanup](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/equation_big.py): `mineru_vl_utils/post_process/equation_big.py`
- [Chart repetition cleanup](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/chart_invalid_repetition.py): `mineru_vl_utils/post_process/chart_invalid_repetition.py`
- [Mermaid cleanup](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/post_process/mermaid_repetition.py): `mineru_vl_utils/post_process/mermaid_repetition.py`

### Shared contracts

- [VLM client contract](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/base_client.py): `mineru_vl_utils/vlm_client/base_client.py`
- [Image utilities](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/vlm_client/utils.py): `mineru_vl_utils/vlm_client/utils.py`
- [No-repeat sampling](https://github.com/opendatalab/mineru-vl-utils/blob/main/mineru_vl_utils/logits_processor/vllm_v1_no_repeat_ngram.py): `mineru_vl_utils/logits_processor/vllm_v1_no_repeat_ngram.py`

### Other components

- **Application caller**
- **OpenAI-compatible server**

## Connections

- Application caller → MinerU client: calls (evidence: `README.md`)
- MinerU client → Layout detection: detects layout (evidence: `mineru_vl_utils/mineru_client.py`)
- MinerU client → Region preparation: prepares regions (evidence: `mineru_vl_utils/mineru_client.py`)
- Region preparation → Table image handling: masks table images (evidence: `mineru_vl_utils/mineru_client.py`)
- MinerU client → HTTP backend: selects backend (evidence: `README.md`)
- MinerU client → Transformers backend: selects backend (evidence: `README.md`)
- MinerU client → MLX backend: selects backend (evidence: `README.md`)
- MinerU client → LMDeploy backend: selects backend (evidence: `README.md`)
- MinerU client → vLLM backend: selects backend (evidence: `README.md`)
- MinerU client → Async vLLM backend: selects backend (evidence: `README.md`)
- MinerU client → llama.cpp backend: selects backend (evidence: `README.md`)
- MinerU client → Block post-processing: processes blocks (evidence: `mineru_vl_utils/mineru_client.py`)
- Block post-processing → Content blocks: updates blocks (evidence: `mineru_vl_utils/post_process/__init__.py`)
- Block post-processing → Image analysis cleanup: analyzes images (evidence: `mineru_vl_utils/post_process/__init__.py`)
- Block post-processing → Table image handling: replaces tokens (evidence: `mineru_vl_utils/post_process/__init__.py`)
- Block post-processing → Equation cleanup: fixes equations (evidence: `mineru_vl_utils/post_process/__init__.py`)
- Image analysis cleanup → Chart repetition cleanup: cleans repetitions (evidence: `mineru_vl_utils/post_process/image_analysis_postprocess.py`)
- Image analysis cleanup → Mermaid cleanup: cleans diagrams (evidence: `mineru_vl_utils/post_process/image_analysis_postprocess.py`)
- MinerU client → Cross-page tables: handles page tables (evidence: `mineru_vl_utils/mineru_client.py`)
- HTTP backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/http_client.py`)
- Transformers backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/base_client.py`)
- MLX backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/base_client.py`)
- LMDeploy backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/lmdeploy_engine_client.py`)
- vLLM backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/vllm_engine_client.py`)
- Async vLLM backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/vllm_async_engine_client.py`)
- llama.cpp backend → VLM client contract: implements contract (evidence: `mineru_vl_utils/vlm_client/llama_cpp_engine_client.py`)
- HTTP backend → Image utilities: loads images (evidence: `mineru_vl_utils/vlm_client/http_client.py`)
- vLLM backend → No-repeat sampling: supports sampling (evidence: `README.md`)
- HTTP backend → OpenAI-compatible server: sends requests (evidence: `README.md`)
- MinerU client → Content blocks: returns results (evidence: `README.md`)
