How opendcai/dataflow works
DataFlow turns user-provided or sourced data into LLM-ready datasets through configurable, reusable operator pipelines: generate or extract content, refine and filter it, then evaluate the resulting data. Users can run predefined workflows from the CLI or build them visually in the WebUI. The core runtime compiles and executes operator sequences over shared storage, with model-serving backends available to operators. The tree and README establish broad capabilities; source excerpts confirm selected execution and storage behavior, so links for unsampled integrations are intentionally omitted.
Main entry points: operators Generation operators · serving Model serving · operator_pipeline.py Visual builder · pipelines PDF and knowledge
Components 20
User interfaces
- CLI
dataflow/cli.py - Visual builder
dataflow/webui/operator_pipeline.py
Pipeline runtime
- Pipeline engine
dataflow/pipeline/Pipeline.py - Operator API
dataflow/core/operator.py - Operator registry
dataflow/utils/registry.py
Data preparation
- Generation operators
dataflow/operators - Refinement operators
dataflow/operators - Filtering operators
dataflow/operators - Evaluation operators
dataflow/operators
Domain workflows
- PDF and knowledge
dataflow/statics/pipelines - Text and QA
dataflow/statics/pipelines - Reasoning synthesis
dataflow/statics/pipelines - Code datasets
dataflow/statics/pipelines - Text-to-SQL
dataflow/statics/pipelines - Agentic RAG
dataflow/statics/pipelines
Runtime services
- Pipeline storage
dataflow/utils/storage.py - Model serving
dataflow/serving
Other components
- Data practitioner
- Prepared datasets
- SQL databases
Connections 23
- Data practitioner → CLI: starts workflow
- Data practitioner → Visual builder: configures workflow
- CLI → Pipeline engine: runs pipeline
- Visual builder → Operator registry: discovers operators
- Visual builder → Pipeline storage: creates storage
- Pipeline engine → Operator API: executes operators
- Operator API → Pipeline storage: reads and writes
- Generation operators → Model serving: requests inference
- Refinement operators → Model serving: requests inference
- Filtering operators → Model serving: requests inference
- Evaluation operators → Model serving: requests inference
- PDF and knowledge → Generation operators: extracts and generates
- Text and QA → Generation operators: generates QA
- Reasoning synthesis → Generation operators: generates answers
- Code datasets → Generation operators: synthesizes code
- Text-to-SQL → SQL databases: executes SQL
- Agentic RAG → Generation operators: generates QA
- PDF and knowledge → Prepared datasets: produces datasets
- Text and QA → Prepared datasets: produces datasets
- Reasoning synthesis → Prepared datasets: produces datasets
- Code datasets → Prepared datasets: produces datasets
- Text-to-SQL → Prepared datasets: produces datasets
- Agentic RAG → Prepared datasets: produces datasets