# ibm/docling-pipelines architecture

> Architecture diagram of the GitHub repository ibm/docling-pipelines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibm/docling-pipelines
- Repository: https://github.com/ibm/docling-pipelines

## Overview

Docling pipelines is a document-curation system for RAG: users define and run DAG flows through the CLI, Python API, or REST interface; flows ingest and extract documents, apply curation and quality operators, then write outputs and embeddings to storage targets. The graph groups execution, pipeline stages, integrations/state, and interfaces. README statements establish the product workflow and capabilities; several implementation directories were not sampled, so their internal call relationships are left unasserted.

Main entry points: [`src/docpipe/core/orchestration/prefect/prefect_engine.py`](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/core/orchestration/prefect/prefect_engine.py) (Prefect engine), [`src/docpipe/api/main.py`](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/main.py) (FastAPI application), [`src/docpipe/lib/docpipe_flow_manager.py`](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/lib/docpipe_flow_manager.py) (Flow manager), [`src/docpipe/cli/docpipe_cli.py`](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/cli/docpipe_cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_cli["CLI<br/>[docpipe_cli.py]"]
  node_python["Python API"]
  node_frontend["Web authoring UI<br/>[Canvas.tsx]"]
end

subgraph group_execution["Flow execution"]
  node_manager["Flow manager"]
  node_engine["Prefect engine<br/>[prefect_engine.py]"]
end

subgraph group_pipeline["Document pipeline"]
  node_ingest["Source ingestion"]
  node_extract["Document extraction"]
  node_curation["Curation operators"]
  node_chunk_embed["Chunking and embeddings"]
end

subgraph group_integrations["Stores and integrations"]
  node_vector["Vector database output"]
  node_storage_output["Storage output"]
  node_job_stats[("Run statistics")]
end

subgraph group_api["REST services"]
  node_backend["FastAPI application<br/>[main.py]"]
  node_flow_api["Flow endpoints<br/>[flows.py]"]
  node_job_api["Run endpoints<br/>[job_runs.py]"]
end

node_user(("Pipeline author"))
node_external_sources["Document sources"]
node_external_services["LLM and extraction services"]
node_targets[("Storage targets")]

node_user -->|"runs flows"| node_cli
node_user -->|"runs flows"| node_python
node_user -.->|"authors flows"| node_frontend
node_frontend -->|"calls REST API"| node_backend
node_backend -.->|"routes flows"| node_flow_api
node_backend -.->|"routes runs"| node_job_api
node_cli -->|"executes flow"| node_manager
node_python -->|"executes flow"| node_manager
node_manager -.->|"dispatches execution"| node_engine
node_engine -->|"executes DAG"| node_ingest
node_engine -->|"executes DAG"| node_extract
node_engine -->|"executes DAG"| node_curation
node_engine -->|"executes DAG"| node_chunk_embed
node_engine -->|"executes DAG"| node_vector
node_engine -->|"executes DAG"| node_storage_output
node_external_sources -->|"provides documents"| node_ingest
node_extract -->|"uses Docling"| node_external_services
node_curation -->|"uses LLMs"| node_external_services
node_chunk_embed -->|"generates embeddings"| node_external_services
node_vector -->|"writes vectors"| node_targets
node_storage_output -->|"writes documents"| node_targets
node_engine -.->|"records run stats"| node_job_stats

click node_cli "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/cli/docpipe_cli.py"
click node_python "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/lib/docpipe_flow_manager.py"
click node_frontend "https://github.com/ibm/docling-pipelines/blob/main/frontend/src/pages/Canvas/Canvas.tsx"
click node_backend "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/main.py"
click node_flow_api "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/routes/flows.py"
click node_job_api "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/routes/job_runs.py"
click node_manager "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/lib/docpipe_flow_manager.py"
click node_engine "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/core/orchestration/prefect/prefect_engine.py"
click node_ingest "https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators"
click node_extract "https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators"
click node_curation "https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators"
click node_chunk_embed "https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators"
click node_vector "https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators/vectordb"
click node_storage_output "https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/core/operators/storage/storage_output_operator.py"
click node_job_stats "https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/job_management"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_python,node_frontend toneBlue
class node_manager,node_engine,node_targets toneAmber
class node_ingest,node_extract,node_curation,node_chunk_embed toneMint
class node_vector,node_storage_output,node_job_stats toneRose
class node_backend,node_flow_api,node_job_api,node_user,node_external_sources,node_external_services toneIndigo
```

## Components

### User interfaces

- [CLI](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/cli/docpipe_cli.py): `src/docpipe/cli/docpipe_cli.py`
- [Python API](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/lib/docpipe_flow_manager.py): `src/docpipe/lib/docpipe_flow_manager.py`
- [Web authoring UI](https://github.com/ibm/docling-pipelines/blob/main/frontend/src/pages/Canvas/Canvas.tsx): `frontend/src/pages/Canvas/Canvas.tsx`

### Flow execution

- [Flow manager](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/lib/docpipe_flow_manager.py): `src/docpipe/lib/docpipe_flow_manager.py`
- [Prefect engine](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/core/orchestration/prefect/prefect_engine.py): `src/docpipe/core/orchestration/prefect/prefect_engine.py`

### Document pipeline

- [Source ingestion](https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators): `src/docpipe/core/operators`
- [Document extraction](https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators): `src/docpipe/core/operators`
- [Curation operators](https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators): `src/docpipe/core/operators`
- [Chunking and embeddings](https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators): `src/docpipe/core/operators`

### Stores and integrations

- [Vector database output](https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/operators/vectordb): `src/docpipe/core/operators/vectordb`
- [Storage output](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/core/operators/storage/storage_output_operator.py): `src/docpipe/core/operators/storage/storage_output_operator.py`
- [Run statistics](https://github.com/ibm/docling-pipelines/tree/main/src/docpipe/core/job_management): `src/docpipe/core/job_management`

### REST services

- [FastAPI application](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/main.py): `src/docpipe/api/main.py`
- [Flow endpoints](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/routes/flows.py): `src/docpipe/api/routes/flows.py`
- [Run endpoints](https://github.com/ibm/docling-pipelines/blob/main/src/docpipe/api/routes/job_runs.py): `src/docpipe/api/routes/job_runs.py`

### Other components

- **Pipeline author**
- **Document sources**
- **LLM and extraction services**
- **Storage targets**

## Connections

- Pipeline author → CLI: runs flows (evidence: `README.md`)
- Pipeline author → Python API: runs flows (evidence: `README.md`)
- Pipeline author → Web authoring UI: authors flows
- Web authoring UI → FastAPI application: calls REST API
- FastAPI application → Flow endpoints: routes flows
- FastAPI application → Run endpoints: routes runs
- CLI → Flow manager: executes flow (evidence: `README.md`)
- Python API → Flow manager: executes flow (evidence: `README.md`)
- Flow manager → Prefect engine: dispatches execution
- Prefect engine → Source ingestion: executes DAG (evidence: `README.md`)
- Prefect engine → Document extraction: executes DAG (evidence: `README.md`)
- Prefect engine → Curation operators: executes DAG (evidence: `README.md`)
- Prefect engine → Chunking and embeddings: executes DAG (evidence: `README.md`)
- Prefect engine → Vector database output: executes DAG (evidence: `README.md`)
- Prefect engine → Storage output: executes DAG (evidence: `README.md`)
- Document sources → Source ingestion: provides documents (evidence: `README.md`)
- Document extraction → LLM and extraction services: uses Docling (evidence: `README.md`)
- Curation operators → LLM and extraction services: uses LLMs (evidence: `README.md`)
- Chunking and embeddings → LLM and extraction services: generates embeddings (evidence: `README.md`)
- Vector database output → Storage targets: writes vectors (evidence: `README.md`)
- Storage output → Storage targets: writes documents (evidence: `README.md`)
- Prefect engine → Run statistics: records run stats
