# x-plug/mplug-docowl architecture

> Architecture diagram of the GitHub repository x-plug/mplug-docowl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/x-plug/mplug-docowl
- Repository: https://github.com/x-plug/mplug-docowl

## Overview

This repository is a family of OCR-free multimodal language-model projects for documents, charts, scientific diagrams, and visually situated language tasks. The typical user workflow is to submit an image and prompt through a demo, prepare image/text inputs, generate a model response, and optionally evaluate it; separate training pipelines prepare datasets and fine-tune models. The graph preserves the README’s major product branches. Some subsystem links are documented only at README level or are omitted where sampled source does not establish the runtime connection.

Main entry points: [`DocOwl1.5/model_worker.py`](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/model_worker.py) (Generation Worker), [`DocOwl1.5/app.py`](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/app.py) (DocOwl Demo), [`UReader/serve/model_worker.py`](https://github.com/x-plug/mplug-docowl/blob/main/UReader/serve/model_worker.py) (Model Worker), [`UReader/mplug_owl/processing_mplug_owl.py`](https://github.com/x-plug/mplug-docowl/blob/main/UReader/mplug_owl/processing_mplug_owl.py) (Image Processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_docowl["DocOwl Models"]
  node_docowl_demo["DocOwl Demo<br/>[app.py]"]
  node_docowl_worker["Generation Worker<br/>[model_worker.py]"]
  node_docowl_conversation["Conversation Prompts<br/>[conversation.py]"]
  node_docowl_image["Image Utilities<br/>[mm_utils.py]"]
  node_docowl_processor["Document Processor<br/>[processor.py]"]
  node_docowl_model["Document Model"]
  node_docowl_training["Fine-tuning<br/>[train_docowl.py]"]
end

subgraph group_ureader["UReader"]
  node_ureader_demo["UReader Demo<br/>[app.py]"]
  node_ureader_server["Demo Server<br/>[web_server.py]"]
  node_ureader_worker["Model Worker<br/>[model_worker.py]"]
  node_ureader_model["Multimodal Model"]
  node_ureader_processor["Image Processor"]
  node_ureader_data["Dataset Pipeline<br/>[xgpt3_dataset.py]"]
  node_ureader_transforms["Document Transforms<br/>[doc_processor.py]"]
  node_ureader_training["Training Pipeline<br/>[train.py]"]
end

subgraph group_chart["TinyChart"]
  node_chart_demo["Chart Demo<br/>[app.py]"]
  node_chart_model["Chart Model<br/>[llava_arch.py]"]
  node_chart_evaluation["Chart Evaluation"]
end

subgraph group_paperowl["PaperOwl"]
  node_paperowl_model["Diagram Model"]
  node_paperowl_data["Diagram Dataset<br/>[xgpt3_dataset.py]"]
  node_paperowl_training["Training Pipeline<br/>[train.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_docowl2_eval["DocOwl2 Evaluation<br/>[dude_eval.py]"]
  node_benchmark_eval["Benchmark Evaluation"]
  node_due_scorers["DUE Scorers<br/>[__init__.py]"]
end

node_user(("User"))

node_user -->|"submits prompt"| node_docowl_demo
node_docowl_demo -->|"requests generation"| node_docowl_worker
node_docowl_worker -->|"processes images"| node_docowl_processor
node_docowl_worker -->|"tokenizes prompt"| node_docowl_image
node_docowl_worker -.->|"generates response"| node_docowl_model
node_docowl_demo -->|"formats dialogue"| node_docowl_conversation
node_docowl_training -->|"prepares images"| node_docowl_processor
node_ureader_demo -->|"starts server"| node_ureader_server
node_ureader_server -->|"serves requests"| node_ureader_worker
node_ureader_worker -->|"runs inference"| node_ureader_model
node_ureader_worker -->|"processes images"| node_ureader_processor
node_ureader_processor -.->|"transforms images"| node_ureader_transforms
node_ureader_training -->|"loads dataset"| node_ureader_data
node_ureader_data -.->|"builds samples"| node_ureader_transforms
node_ureader_processor -.->|"supplies model inputs"| node_ureader_model
node_user -->|"submits prompt"| node_ureader_demo
node_user -->|"asks chart question"| node_chart_demo
node_chart_demo -.->|"runs generation"| node_chart_model
node_chart_demo -->|"reports metrics"| node_chart_evaluation
node_paperowl_training -->|"loads data"| node_paperowl_data
node_paperowl_data -.->|"supports training"| node_paperowl_model
node_benchmark_eval -->|"dispatches evaluation"| node_docowl2_eval
node_docowl2_eval -.->|"uses scorers"| node_due_scorers
node_user -->|"analyzes diagrams"| node_paperowl_model
node_user -->|"evaluates results"| node_docowl2_eval

click node_docowl_demo "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/app.py"
click node_docowl_worker "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/model_worker.py"
click node_docowl_conversation "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/conversation.py"
click node_docowl_image "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/mm_utils.py"
click node_docowl_processor "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/processor.py"
click node_docowl_model "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/model/modeling_mplug_docowl.py"
click node_docowl_training "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/train/train_docowl.py"
click node_ureader_demo "https://github.com/x-plug/mplug-docowl/blob/main/UReader/app.py"
click node_ureader_server "https://github.com/x-plug/mplug-docowl/blob/main/UReader/serve/web_server.py"
click node_ureader_worker "https://github.com/x-plug/mplug-docowl/blob/main/UReader/serve/model_worker.py"
click node_ureader_model "https://github.com/x-plug/mplug-docowl/blob/main/UReader/mplug_owl/modeling_mplug_owl.py"
click node_ureader_processor "https://github.com/x-plug/mplug-docowl/blob/main/UReader/mplug_owl/processing_mplug_owl.py"
click node_ureader_data "https://github.com/x-plug/mplug-docowl/blob/main/UReader/pipeline/data_utils/xgpt3_dataset.py"
click node_ureader_transforms "https://github.com/x-plug/mplug-docowl/blob/main/UReader/pipeline/data_utils/processors/doc_processor.py"
click node_ureader_training "https://github.com/x-plug/mplug-docowl/blob/main/UReader/pipeline/train.py"
click node_chart_demo "https://github.com/x-plug/mplug-docowl/blob/main/TinyChart/app.py"
click node_chart_model "https://github.com/x-plug/mplug-docowl/blob/main/TinyChart/tinychart/model/llava_arch.py"
click node_chart_evaluation "https://github.com/x-plug/mplug-docowl/blob/main/TinyChart/tinychart/eval/eval_chart2table.py"
click node_paperowl_model "https://github.com/x-plug/mplug-docowl/blob/main/PaperOwl/mplug_owl/modeling_mplug_owl.py"
click node_paperowl_data "https://github.com/x-plug/mplug-docowl/blob/main/PaperOwl/pipeline/data_utils/xgpt3_dataset.py"
click node_paperowl_training "https://github.com/x-plug/mplug-docowl/blob/main/PaperOwl/pipeline/train.py"
click node_docowl2_eval "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl2/evaluation/dude_eval.py"
click node_benchmark_eval "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl2/docowl_benchmark_evaluate.py"
click node_due_scorers "https://github.com/x-plug/mplug-docowl/blob/main/DocOwl2/evaluation/due_evaluator/scorers/__init__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_docowl_demo,node_docowl_worker,node_docowl_conversation,node_docowl_image,node_docowl_processor,node_docowl_model,node_docowl_training,node_user toneBlue
class node_ureader_demo,node_ureader_server,node_ureader_worker,node_ureader_model,node_ureader_processor,node_ureader_data,node_ureader_transforms,node_ureader_training toneAmber
class node_chart_demo,node_chart_model,node_chart_evaluation toneMint
class node_paperowl_model,node_paperowl_data,node_paperowl_training toneRose
class node_docowl2_eval,node_benchmark_eval,node_due_scorers toneIndigo
```

## Components

### DocOwl Models

- [DocOwl Demo](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/app.py): `DocOwl1.5/app.py`
- [Generation Worker](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/model_worker.py): `DocOwl1.5/model_worker.py`
- [Conversation Prompts](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/conversation.py): `DocOwl1.5/mplug_docowl/conversation.py`
- [Image Utilities](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/mm_utils.py): `DocOwl1.5/mplug_docowl/mm_utils.py`
- [Document Processor](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/processor.py): `DocOwl1.5/mplug_docowl/processor.py`
- [Document Model](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/model/modeling_mplug_docowl.py): `DocOwl1.5/mplug_docowl/model/modeling_mplug_docowl.py`
- [Fine-tuning](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl1.5/mplug_docowl/train/train_docowl.py): `DocOwl1.5/mplug_docowl/train/train_docowl.py`

### UReader

- [UReader Demo](https://github.com/x-plug/mplug-docowl/blob/main/UReader/app.py): `UReader/app.py`
- [Demo Server](https://github.com/x-plug/mplug-docowl/blob/main/UReader/serve/web_server.py): `UReader/serve/web_server.py`
- [Model Worker](https://github.com/x-plug/mplug-docowl/blob/main/UReader/serve/model_worker.py): `UReader/serve/model_worker.py`
- [Multimodal Model](https://github.com/x-plug/mplug-docowl/blob/main/UReader/mplug_owl/modeling_mplug_owl.py): `UReader/mplug_owl/modeling_mplug_owl.py`
- [Image Processor](https://github.com/x-plug/mplug-docowl/blob/main/UReader/mplug_owl/processing_mplug_owl.py): `UReader/mplug_owl/processing_mplug_owl.py`
- [Dataset Pipeline](https://github.com/x-plug/mplug-docowl/blob/main/UReader/pipeline/data_utils/xgpt3_dataset.py): `UReader/pipeline/data_utils/xgpt3_dataset.py`
- [Document Transforms](https://github.com/x-plug/mplug-docowl/blob/main/UReader/pipeline/data_utils/processors/doc_processor.py): `UReader/pipeline/data_utils/processors/doc_processor.py`
- [Training Pipeline](https://github.com/x-plug/mplug-docowl/blob/main/UReader/pipeline/train.py): `UReader/pipeline/train.py`

### TinyChart

- [Chart Demo](https://github.com/x-plug/mplug-docowl/blob/main/TinyChart/app.py): `TinyChart/app.py`
- [Chart Model](https://github.com/x-plug/mplug-docowl/blob/main/TinyChart/tinychart/model/llava_arch.py): `TinyChart/tinychart/model/llava_arch.py`
- [Chart Evaluation](https://github.com/x-plug/mplug-docowl/blob/main/TinyChart/tinychart/eval/eval_chart2table.py): `TinyChart/tinychart/eval/eval_chart2table.py`

### PaperOwl

- [Diagram Model](https://github.com/x-plug/mplug-docowl/blob/main/PaperOwl/mplug_owl/modeling_mplug_owl.py): `PaperOwl/mplug_owl/modeling_mplug_owl.py`
- [Diagram Dataset](https://github.com/x-plug/mplug-docowl/blob/main/PaperOwl/pipeline/data_utils/xgpt3_dataset.py): `PaperOwl/pipeline/data_utils/xgpt3_dataset.py`
- [Training Pipeline](https://github.com/x-plug/mplug-docowl/blob/main/PaperOwl/pipeline/train.py): `PaperOwl/pipeline/train.py`

### Evaluation

- [DocOwl2 Evaluation](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl2/evaluation/dude_eval.py): `DocOwl2/evaluation/dude_eval.py`
- [Benchmark Evaluation](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl2/docowl_benchmark_evaluate.py): `DocOwl2/docowl_benchmark_evaluate.py`
- [DUE Scorers](https://github.com/x-plug/mplug-docowl/blob/main/DocOwl2/evaluation/due_evaluator/scorers/__init__.py): `DocOwl2/evaluation/due_evaluator/scorers/__init__.py`

### Other components

- **User**

## Connections

- User → DocOwl Demo: submits prompt
- DocOwl Demo → Generation Worker: requests generation (evidence: `DocOwl1.5/app.py`)
- Generation Worker → Document Processor: processes images (evidence: `DocOwl1.5/model_worker.py`)
- Generation Worker → Image Utilities: tokenizes prompt (evidence: `DocOwl1.5/model_worker.py`)
- Generation Worker → Document Model: generates response
- DocOwl Demo → Conversation Prompts: formats dialogue (evidence: `DocOwl1.5/app.py`)
- Fine-tuning → Document Processor: prepares images (evidence: `DocOwl1.5/mplug_docowl/train/train_docowl.py`)
- UReader Demo → Demo Server: starts server (evidence: `UReader/app.py`)
- Demo Server → Model Worker: serves requests (evidence: `UReader/serve/web_server.py`)
- Model Worker → Multimodal Model: runs inference (evidence: `UReader/serve/model_worker.py`)
- Model Worker → Image Processor: processes images (evidence: `UReader/serve/model_worker.py`)
- Image Processor → Document Transforms: transforms images
- Training Pipeline → Dataset Pipeline: loads dataset (evidence: `UReader/pipeline/train.py`)
- Dataset Pipeline → Document Transforms: builds samples
- Image Processor → Multimodal Model: supplies model inputs
- User → UReader Demo: submits prompt
- User → Chart Demo: asks chart question
- Chart Demo → Chart Model: runs generation
- Chart Demo → Chart Evaluation: reports metrics (evidence: `TinyChart/app.py`)
- Training Pipeline → Diagram Dataset: loads data (evidence: `PaperOwl/pipeline/train.py`)
- Diagram Dataset → Diagram Model: supports training
- Benchmark Evaluation → DocOwl2 Evaluation: dispatches evaluation (evidence: `DocOwl2/docowl_benchmark_evaluate.py`)
- DocOwl2 Evaluation → DUE Scorers: uses scorers
- User → Diagram Model: analyzes diagrams
- User → DocOwl2 Evaluation: evaluates results