How x-plug/mplug-docowl works
This repository is a family of OCR-free multimodal language-model projects for documents, charts, scientific diagrams, and visually situated language tasks. The typical user workflow is to submit an image and prompt through a demo, prepare image/text inputs, generate a model response, and optionally evaluate it; separate training pipelines prepare datasets and fine-tune models. The graph preserves the README’s major product branches. Some subsystem links are documented only at README level or are omitted where sampled source does not establish the runtime connection.
Main entry points: DocOwl1.5/model_worker.py Generation Worker · app.py DocOwl Demo · serve/model_worker.py Model Worker · processing_mplug_owl.py Image Processor
Components 25
DocOwl Models
- DocOwl Demo
DocOwl1.5/app.py - Generation Worker
DocOwl1.5/model_worker.py - Conversation Prompts
DocOwl1.5/mplug_docowl/conversation.py - Image Utilities
DocOwl1.5/mplug_docowl/mm_utils.py - Document Processor
DocOwl1.5/mplug_docowl/processor.py - Document Model
DocOwl1.5/mplug_docowl/model/modeling_mplug_docowl.py - Fine-tuning
DocOwl1.5/mplug_docowl/train/train_docowl.py
UReader
- UReader Demo
UReader/app.py - Demo Server
UReader/serve/web_server.py - Model Worker
UReader/serve/model_worker.py - Multimodal Model
UReader/mplug_owl/modeling_mplug_owl.py - Image Processor
UReader/mplug_owl/processing_mplug_owl.py - Dataset Pipeline
UReader/pipeline/data_utils/xgpt3_dataset.py - Document Transforms
UReader/pipeline/data_utils/processors/doc_processor.py - Training Pipeline
UReader/pipeline/train.py
TinyChart
- Chart Demo
TinyChart/app.py - Chart Model
TinyChart/tinychart/model/llava_arch.py - Chart Evaluation
TinyChart/tinychart/eval/eval_chart2table.py
PaperOwl
- Diagram Model
PaperOwl/mplug_owl/modeling_mplug_owl.py - Diagram Dataset
PaperOwl/pipeline/data_utils/xgpt3_dataset.py - Training Pipeline
PaperOwl/pipeline/train.py
Evaluation
- DocOwl2 Evaluation
DocOwl2/evaluation/dude_eval.py - Benchmark Evaluation
DocOwl2/docowl_benchmark_evaluate.py - DUE Scorers
DocOwl2/evaluation/due_evaluator/scorers/__init__.py
Other components
- User
Connections 25
- User → DocOwl Demo: submits prompt
- DocOwl Demo → Generation Worker: requests generation
DocOwl1.5/app.py - Generation Worker → Document Processor: processes images
DocOwl1.5/model_worker.py - Generation Worker → Image Utilities: tokenizes prompt
DocOwl1.5/model_worker.py - Generation Worker → Document Model: generates response
- DocOwl Demo → Conversation Prompts: formats dialogue
DocOwl1.5/app.py - Fine-tuning → Document Processor: prepares images
DocOwl1.5/mplug_docowl/train/train_docowl.py - UReader Demo → Demo Server: starts server
UReader/app.py - Demo Server → Model Worker: serves requests
UReader/serve/web_server.py - Model Worker → Multimodal Model: runs inference
UReader/serve/model_worker.py - Model Worker → Image Processor: processes images
UReader/serve/model_worker.py - Image Processor → Document Transforms: transforms images
- Training Pipeline → Dataset Pipeline: loads dataset
UReader/pipeline/train.py - Dataset Pipeline → Document Transforms: builds samples
- Image Processor → Multimodal Model: supplies model inputs
- User → UReader Demo: submits prompt
- User → Chart Demo: asks chart question
- Chart Demo → Chart Model: runs generation
- Chart Demo → Chart Evaluation: reports metrics
TinyChart/app.py - Training Pipeline → Diagram Dataset: loads data
PaperOwl/pipeline/train.py - Diagram Dataset → Diagram Model: supports training
- Benchmark Evaluation → DocOwl2 Evaluation: dispatches evaluation
DocOwl2/docowl_benchmark_evaluate.py - DocOwl2 Evaluation → DUE Scorers: uses scorers
- User → Diagram Model: analyzes diagrams
- User → DocOwl2 Evaluation: evaluates results