# zi-deng/flow-dc architecture

> Architecture diagram of the GitHub repository zi-deng/flow-dc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zi-deng/flow-dc
- Repository: https://github.com/zi-deng/flow-dc

## Overview

FLOW-DC is a distributed dataset-acquisition system centered on TaskVine downloads and PAARC per-host congestion control. Its principal workflow partitions a dataset, assigns download work to workers, and consolidates outputs; local batch downloading and output integrity are supporting paths. The repository also documents bounded Jetstream2 pilot operations, experiment preparation and execution, and an agentic issue-to-review workflow. Relationships below are limited to README statements and SOURCE INDEX references; unsampled implementation details are not inferred.

Main entry points: [`bin/download_batch.py`](https://github.com/zi-deng/flow-dc/blob/main/bin/download_batch.py) (Batch downloader), [`bin/flowdc_pilot_cli.py`](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_cli.py) (Pilot lifecycle), [`scripts/agentic/workflow.py`](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/workflow.py) (Workflow coordinator), [`bin/flowdc_vine.py`](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine.py) (Vine runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Dataset acquisition"]
  node_manifest_partitioning["Manifest partitioning<br/>[SplitParquet.py]"]
  node_batch_downloader["Batch downloader<br/>[download_batch.py]"]
  node_http_download["HTTP acquisition<br/>[single_download.py]"]
  node_paarc["PAARC control<br/>[download_batch.py]"]
  node_output_integrity["Output integrity"]
end

subgraph group_distributed["Distributed execution"]
  node_taskvine_manager["TaskVine manager<br/>[TaskvineFLOWDC.py]"]
  node_taskvine_cloud["Cloud manager"]
  node_vine_runtime["Vine runtime<br/>[flowdc_vine.py]"]
  node_vine_worker["Vine worker"]
  node_vine_protocol["Vine protocol"]
end

subgraph group_operations["Pilot operations"]
  node_jetstream_ops["Operations CLI<br/>[flowdc_ops.py]"]
  node_topology["VM topology<br/>[flowdc_topology.py]"]
  node_pilot_cli["Pilot lifecycle"]
  node_pilot_journal[("Pilot journal")]
  node_pilot_provider["Cloud provider"]
  node_pilot_supervisor["Shutdown supervisor"]
end

subgraph group_experiments["Research experiments"]
  node_experiment_runner["Experiment runner"]
  node_experiment_data["Experiment data"]
  node_experiment_research["Research bridge"]
end

subgraph group_agentic["Agentic workflow"]
  node_workflow_coordinator["Workflow coordinator<br/>[workflow.py]"]
  node_task_registry[("Task registry<br/>[tasks.py]")]
  node_managed_sessions["Managed sessions<br/>[sessions.py]"]
  node_review_pipeline["Review pipeline<br/>[pipeline.py]"]
  node_independent_review["Independent review<br/>[review.py]"]
  node_coverage_gate["Coverage gate<br/>[review_coverage.py]"]
end

node_researcher(("Researcher"))

node_researcher -->|"partitions manifest"| node_manifest_partitioning
node_researcher -->|"starts download"| node_batch_downloader
node_batch_downloader -->|"controls concurrency"| node_paarc
node_batch_downloader -->|"performs requests"| node_http_download
node_batch_downloader -->|"uses integrity"| node_output_integrity
node_researcher -->|"starts distributed run"| node_taskvine_manager
node_taskvine_manager -->|"uses runtime"| node_vine_runtime
node_vine_runtime -->|"dispatches tasks"| node_vine_worker
node_vine_worker -->|"runs downloads"| node_batch_downloader
node_vine_runtime -->|"uses protocol"| node_vine_protocol
node_researcher -.->|"starts cloud run"| node_taskvine_cloud
node_researcher -->|"runs operations"| node_jetstream_ops
node_jetstream_ops -->|"uses topology"| node_topology
node_jetstream_ops -->|"dispatches pilot"| node_pilot_cli
node_pilot_cli -->|"records state"| node_pilot_journal
node_pilot_cli -->|"uses provider"| node_pilot_provider
node_pilot_cli -->|"controls supervisor"| node_pilot_supervisor
node_researcher -->|"runs experiment"| node_experiment_runner
node_experiment_runner -->|"prepares inputs"| node_experiment_data
node_experiment_runner -->|"uses research bridge"| node_experiment_research
node_researcher -->|"starts task workflow"| node_workflow_coordinator
node_workflow_coordinator -->|"registers tasks"| node_task_registry
node_workflow_coordinator -->|"manages execution"| node_managed_sessions
node_workflow_coordinator -->|"coordinates review"| node_review_pipeline
node_review_pipeline -->|"invokes review"| node_independent_review
node_independent_review -->|"checks coverage"| node_coverage_gate

click node_manifest_partitioning "https://github.com/zi-deng/flow-dc/blob/main/bin/SplitParquet.py"
click node_batch_downloader "https://github.com/zi-deng/flow-dc/blob/main/bin/download_batch.py"
click node_http_download "https://github.com/zi-deng/flow-dc/blob/main/bin/single_download.py"
click node_paarc "https://github.com/zi-deng/flow-dc/blob/main/bin/download_batch.py"
click node_output_integrity "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_integrity.py"
click node_taskvine_manager "https://github.com/zi-deng/flow-dc/blob/main/bin/TaskvineFLOWDC.py"
click node_taskvine_cloud "https://github.com/zi-deng/flow-dc/blob/main/bin/TaskvineFLOWDCCloud.py"
click node_vine_runtime "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine.py"
click node_vine_worker "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine_worker.py"
click node_vine_protocol "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine_protocol.py"
click node_jetstream_ops "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_ops.py"
click node_topology "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_topology.py"
click node_pilot_cli "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_cli.py"
click node_pilot_journal "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_journal.py"
click node_pilot_provider "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_provider.py"
click node_pilot_supervisor "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_supervisor.py"
click node_experiment_runner "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_experiment.py"
click node_experiment_data "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_experiment_data.py"
click node_experiment_research "https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_experiment_research.py"
click node_workflow_coordinator "https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/workflow.py"
click node_task_registry "https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/tasks.py"
click node_managed_sessions "https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/sessions.py"
click node_review_pipeline "https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/pipeline.py"
click node_independent_review "https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/review.py"
click node_coverage_gate "https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/review_coverage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_manifest_partitioning,node_batch_downloader,node_http_download,node_paarc,node_output_integrity toneBlue
class node_taskvine_manager,node_taskvine_cloud,node_vine_runtime,node_vine_worker,node_vine_protocol toneAmber
class node_jetstream_ops,node_topology,node_pilot_cli,node_pilot_journal,node_pilot_provider,node_pilot_supervisor toneMint
class node_experiment_runner,node_experiment_data,node_experiment_research toneRose
class node_workflow_coordinator,node_task_registry,node_managed_sessions,node_review_pipeline,node_independent_review,node_coverage_gate,node_researcher toneIndigo
```

## Components

### Dataset acquisition

- [Manifest partitioning](https://github.com/zi-deng/flow-dc/blob/main/bin/SplitParquet.py): `bin/SplitParquet.py`
- [Batch downloader](https://github.com/zi-deng/flow-dc/blob/main/bin/download_batch.py): `bin/download_batch.py`
- [HTTP acquisition](https://github.com/zi-deng/flow-dc/blob/main/bin/single_download.py): `bin/single_download.py`
- [PAARC control](https://github.com/zi-deng/flow-dc/blob/main/bin/download_batch.py): `bin/download_batch.py`
- [Output integrity](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_integrity.py): `bin/flowdc_integrity.py`

### Distributed execution

- [TaskVine manager](https://github.com/zi-deng/flow-dc/blob/main/bin/TaskvineFLOWDC.py): `bin/TaskvineFLOWDC.py`
- [Cloud manager](https://github.com/zi-deng/flow-dc/blob/main/bin/TaskvineFLOWDCCloud.py): `bin/TaskvineFLOWDCCloud.py`
- [Vine runtime](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine.py): `bin/flowdc_vine.py`
- [Vine worker](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine_worker.py): `bin/flowdc_vine_worker.py`
- [Vine protocol](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_vine_protocol.py): `bin/flowdc_vine_protocol.py`

### Pilot operations

- [Operations CLI](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_ops.py): `bin/flowdc_ops.py`
- [VM topology](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_topology.py): `bin/flowdc_topology.py`
- [Pilot lifecycle](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_cli.py): `bin/flowdc_pilot_cli.py`
- [Pilot journal](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_journal.py): `bin/flowdc_pilot_journal.py`
- [Cloud provider](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_provider.py): `bin/flowdc_pilot_provider.py`
- [Shutdown supervisor](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_pilot_supervisor.py): `bin/flowdc_pilot_supervisor.py`

### Research experiments

- [Experiment runner](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_experiment.py): `bin/flowdc_experiment.py`
- [Experiment data](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_experiment_data.py): `bin/flowdc_experiment_data.py`
- [Research bridge](https://github.com/zi-deng/flow-dc/blob/main/bin/flowdc_experiment_research.py): `bin/flowdc_experiment_research.py`

### Agentic workflow

- [Workflow coordinator](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/workflow.py): `scripts/agentic/workflow.py`
- [Task registry](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/tasks.py): `scripts/agentic/tasks.py`
- [Managed sessions](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/sessions.py): `scripts/agentic/sessions.py`
- [Review pipeline](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/pipeline.py): `scripts/agentic/pipeline.py`
- [Independent review](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/review.py): `scripts/agentic/review.py`
- [Coverage gate](https://github.com/zi-deng/flow-dc/blob/main/scripts/agentic/review_coverage.py): `scripts/agentic/review_coverage.py`

### Other components

- **Researcher**

## Connections

- Researcher → Manifest partitioning: partitions manifest
- Researcher → Batch downloader: starts download
- Batch downloader → PAARC control: controls concurrency (evidence: `bin/download_batch.py`)
- Batch downloader → HTTP acquisition: performs requests (evidence: `bin/download_batch.py`)
- Batch downloader → Output integrity: uses integrity (evidence: `bin/download_batch.py`)
- Researcher → TaskVine manager: starts distributed run
- TaskVine manager → Vine runtime: uses runtime
- Vine runtime → Vine worker: dispatches tasks (evidence: `bin/flowdc_vine.py`)
- Vine worker → Batch downloader: runs downloads (evidence: `bin/flowdc_vine_worker.py`)
- Vine runtime → Vine protocol: uses protocol (evidence: `bin/flowdc_vine.py`)
- Researcher → Cloud manager: starts cloud run
- Researcher → Operations CLI: runs operations
- Operations CLI → VM topology: uses topology (evidence: `bin/flowdc_ops.py`)
- Operations CLI → Pilot lifecycle: dispatches pilot (evidence: `bin/flowdc_ops.py`)
- Pilot lifecycle → Pilot journal: records state (evidence: `bin/flowdc_pilot_cli.py`)
- Pilot lifecycle → Cloud provider: uses provider (evidence: `bin/flowdc_pilot_cli.py`)
- Pilot lifecycle → Shutdown supervisor: controls supervisor (evidence: `bin/flowdc_pilot_cli.py`)
- Researcher → Experiment runner: runs experiment
- Experiment runner → Experiment data: prepares inputs (evidence: `bin/flowdc_experiment.py`)
- Experiment runner → Research bridge: uses research bridge (evidence: `bin/flowdc_experiment.py`)
- Researcher → Workflow coordinator: starts task workflow
- Workflow coordinator → Task registry: registers tasks (evidence: `scripts/agentic/workflow.py`)
- Workflow coordinator → Managed sessions: manages execution (evidence: `scripts/agentic/workflow.py`)
- Workflow coordinator → Review pipeline: coordinates review (evidence: `scripts/agentic/workflow.py`)
- Review pipeline → Independent review: invokes review (evidence: `scripts/agentic/pipeline.py`)
- Independent review → Coverage gate: checks coverage (evidence: `scripts/agentic/review.py`)
