# opendcai/dataflow architecture

> Architecture diagram of the GitHub repository opendcai/dataflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendcai/dataflow
- Repository: https://github.com/opendcai/dataflow

## Overview

DataFlow turns user-provided or sourced data into LLM-ready datasets through configurable, reusable operator pipelines: generate or extract content, refine and filter it, then evaluate the resulting data. Users can run predefined workflows from the CLI or build them visually in the WebUI. The core runtime compiles and executes operator sequences over shared storage, with model-serving backends available to operators. The tree and README establish broad capabilities; source excerpts confirm selected execution and storage behavior, so links for unsampled integrations are intentionally omitted.

Main entry points: [`dataflow/operators`](https://github.com/opendcai/dataflow/tree/main/dataflow/operators) (Generation operators), [`dataflow/serving`](https://github.com/opendcai/dataflow/tree/main/dataflow/serving) (Model serving), [`dataflow/webui/operator_pipeline.py`](https://github.com/opendcai/dataflow/blob/main/dataflow/webui/operator_pipeline.py) (Visual builder), [`dataflow/statics/pipelines`](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines) (PDF and knowledge)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_cli["CLI<br/>[cli.py]"]
  node_webui["Visual builder"]
end

subgraph group_runtime["Pipeline runtime"]
  node_pipeline["Pipeline engine<br/>[Pipeline.py]"]
  node_operator["Operator API<br/>[operator.py]"]
  node_registry["Operator registry<br/>[registry.py]"]
end

subgraph group_preparation["Data preparation"]
  node_generation["Generation operators"]
  node_refinement["Refinement operators"]
  node_filtering["Filtering operators"]
  node_evaluation["Evaluation operators"]
end

subgraph group_workflows["Domain workflows"]
  node_knowledge["PDF and knowledge"]
  node_textqa["Text and QA"]
  node_reasoning["Reasoning synthesis"]
  node_code["Code datasets"]
  node_textsql["Text-to-SQL"]
  node_rag["Agentic RAG"]
end

subgraph group_infrastructure["Runtime services"]
  node_storage[("Pipeline storage<br/>[storage.py]")]
  node_serving["Model serving"]
end

node_user(("Data practitioner"))
node_results["Prepared datasets"]
node_database[("SQL databases")]

node_user -->|"starts workflow"| node_cli
node_user -->|"configures workflow"| node_webui
node_cli -->|"runs pipeline"| node_pipeline
node_webui -->|"discovers operators"| node_registry
node_webui -->|"creates storage"| node_storage
node_pipeline -->|"executes operators"| node_operator
node_operator -->|"reads and writes"| node_storage
node_generation -.->|"requests inference"| node_serving
node_refinement -.->|"requests inference"| node_serving
node_filtering -.->|"requests inference"| node_serving
node_evaluation -.->|"requests inference"| node_serving
node_knowledge -->|"extracts and generates"| node_generation
node_textqa -->|"generates QA"| node_generation
node_reasoning -->|"generates answers"| node_generation
node_code -->|"synthesizes code"| node_generation
node_textsql -->|"executes SQL"| node_database
node_rag -->|"generates QA"| node_generation
node_knowledge -->|"produces datasets"| node_results
node_textqa -->|"produces datasets"| node_results
node_reasoning -->|"produces datasets"| node_results
node_code -->|"produces datasets"| node_results
node_textsql -->|"produces datasets"| node_results
node_rag -->|"produces datasets"| node_results

click node_cli "https://github.com/opendcai/dataflow/blob/main/dataflow/cli.py"
click node_webui "https://github.com/opendcai/dataflow/blob/main/dataflow/webui/operator_pipeline.py"
click node_pipeline "https://github.com/opendcai/dataflow/blob/main/dataflow/pipeline/Pipeline.py"
click node_operator "https://github.com/opendcai/dataflow/blob/main/dataflow/core/operator.py"
click node_registry "https://github.com/opendcai/dataflow/blob/main/dataflow/utils/registry.py"
click node_storage "https://github.com/opendcai/dataflow/blob/main/dataflow/utils/storage.py"
click node_serving "https://github.com/opendcai/dataflow/tree/main/dataflow/serving"
click node_generation "https://github.com/opendcai/dataflow/tree/main/dataflow/operators"
click node_refinement "https://github.com/opendcai/dataflow/tree/main/dataflow/operators"
click node_filtering "https://github.com/opendcai/dataflow/tree/main/dataflow/operators"
click node_evaluation "https://github.com/opendcai/dataflow/tree/main/dataflow/operators"
click node_knowledge "https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines"
click node_textqa "https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines"
click node_reasoning "https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines"
click node_code "https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines"
click node_textsql "https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines"
click node_rag "https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_webui toneBlue
class node_pipeline,node_operator,node_registry,node_database toneAmber
class node_generation,node_refinement,node_filtering,node_evaluation toneMint
class node_knowledge,node_textqa,node_reasoning,node_code,node_textsql,node_rag toneRose
class node_storage,node_serving,node_user,node_results toneIndigo
```

## Components

### User interfaces

- [CLI](https://github.com/opendcai/dataflow/blob/main/dataflow/cli.py): `dataflow/cli.py`
- [Visual builder](https://github.com/opendcai/dataflow/blob/main/dataflow/webui/operator_pipeline.py): `dataflow/webui/operator_pipeline.py`

### Pipeline runtime

- [Pipeline engine](https://github.com/opendcai/dataflow/blob/main/dataflow/pipeline/Pipeline.py): `dataflow/pipeline/Pipeline.py`
- [Operator API](https://github.com/opendcai/dataflow/blob/main/dataflow/core/operator.py): `dataflow/core/operator.py`
- [Operator registry](https://github.com/opendcai/dataflow/blob/main/dataflow/utils/registry.py): `dataflow/utils/registry.py`

### Data preparation

- [Generation operators](https://github.com/opendcai/dataflow/tree/main/dataflow/operators): `dataflow/operators`
- [Refinement operators](https://github.com/opendcai/dataflow/tree/main/dataflow/operators): `dataflow/operators`
- [Filtering operators](https://github.com/opendcai/dataflow/tree/main/dataflow/operators): `dataflow/operators`
- [Evaluation operators](https://github.com/opendcai/dataflow/tree/main/dataflow/operators): `dataflow/operators`

### Domain workflows

- [PDF and knowledge](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines): `dataflow/statics/pipelines`
- [Text and QA](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines): `dataflow/statics/pipelines`
- [Reasoning synthesis](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines): `dataflow/statics/pipelines`
- [Code datasets](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines): `dataflow/statics/pipelines`
- [Text-to-SQL](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines): `dataflow/statics/pipelines`
- [Agentic RAG](https://github.com/opendcai/dataflow/tree/main/dataflow/statics/pipelines): `dataflow/statics/pipelines`

### Runtime services

- [Pipeline storage](https://github.com/opendcai/dataflow/blob/main/dataflow/utils/storage.py): `dataflow/utils/storage.py`
- [Model serving](https://github.com/opendcai/dataflow/tree/main/dataflow/serving): `dataflow/serving`

### Other components

- **Data practitioner**
- **Prepared datasets**
- **SQL databases**

## Connections

- Data practitioner → CLI: starts workflow
- Data practitioner → Visual builder: configures workflow
- CLI → Pipeline engine: runs pipeline
- Visual builder → Operator registry: discovers operators
- Visual builder → Pipeline storage: creates storage
- Pipeline engine → Operator API: executes operators
- Operator API → Pipeline storage: reads and writes
- Generation operators → Model serving: requests inference
- Refinement operators → Model serving: requests inference
- Filtering operators → Model serving: requests inference
- Evaluation operators → Model serving: requests inference
- PDF and knowledge → Generation operators: extracts and generates
- Text and QA → Generation operators: generates QA
- Reasoning synthesis → Generation operators: generates answers
- Code datasets → Generation operators: synthesizes code
- Text-to-SQL → SQL databases: executes SQL
- Agentic RAG → Generation operators: generates QA
- PDF and knowledge → Prepared datasets: produces datasets
- Text and QA → Prepared datasets: produces datasets
- Reasoning synthesis → Prepared datasets: produces datasets
- Code datasets → Prepared datasets: produces datasets
- Text-to-SQL → Prepared datasets: produces datasets
- Agentic RAG → Prepared datasets: produces datasets