How ibm/docling-pipelines works
Docling pipelines is a document-curation system for RAG: users define and run DAG flows through the CLI, Python API, or REST interface; flows ingest and extract documents, apply curation and quality operators, then write outputs and embeddings to storage targets. The graph groups execution, pipeline stages, integrations/state, and interfaces. README statements establish the product workflow and capabilities; several implementation directories were not sampled, so their internal call relationships are left unasserted.
Main entry points: prefect_engine.py Prefect engine · main.py FastAPI application · docpipe_flow_manager.py Flow manager · docpipe_cli.py CLI
Components 19
User interfaces
- CLI
src/docpipe/cli/docpipe_cli.py - Python API
src/docpipe/lib/docpipe_flow_manager.py - Web authoring UI
frontend/src/pages/Canvas/Canvas.tsx
Flow execution
- Flow manager
src/docpipe/lib/docpipe_flow_manager.py - Prefect engine
src/docpipe/core/orchestration/prefect/prefect_engine.py
Document pipeline
- Source ingestion
src/docpipe/core/operators - Document extraction
src/docpipe/core/operators - Curation operators
src/docpipe/core/operators - Chunking and embeddings
src/docpipe/core/operators
Stores and integrations
- Vector database output
src/docpipe/core/operators/vectordb - Storage output
src/docpipe/core/operators/storage/storage_output_operator.py - Run statistics
src/docpipe/core/job_management
REST services
- FastAPI application
src/docpipe/api/main.py - Flow endpoints
src/docpipe/api/routes/flows.py - Run endpoints
src/docpipe/api/routes/job_runs.py
Other components
- Pipeline author
- Document sources
- LLM and extraction services
- Storage targets
Connections 22
- Pipeline author → CLI: runs flows
README.md - Pipeline author → Python API: runs flows
README.md - Pipeline author → Web authoring UI: authors flows
- Web authoring UI → FastAPI application: calls REST API
- FastAPI application → Flow endpoints: routes flows
- FastAPI application → Run endpoints: routes runs
- CLI → Flow manager: executes flow
README.md - Python API → Flow manager: executes flow
README.md - Flow manager → Prefect engine: dispatches execution
- Prefect engine → Source ingestion: executes DAG
README.md - Prefect engine → Document extraction: executes DAG
README.md - Prefect engine → Curation operators: executes DAG
README.md - Prefect engine → Chunking and embeddings: executes DAG
README.md - Prefect engine → Vector database output: executes DAG
README.md - Prefect engine → Storage output: executes DAG
README.md - Document sources → Source ingestion: provides documents
README.md - Document extraction → LLM and extraction services: uses Docling
README.md - Curation operators → LLM and extraction services: uses LLMs
README.md - Chunking and embeddings → LLM and extraction services: generates embeddings
README.md - Vector database output → Storage targets: writes vectors
README.md - Storage output → Storage targets: writes documents
README.md - Prefect engine → Run statistics: records run stats