How zi-deng/flow-dc works
FLOW-DC is a distributed dataset-acquisition system centered on TaskVine downloads and PAARC per-host congestion control. Its principal workflow partitions a dataset, assigns download work to workers, and consolidates outputs; local batch downloading and output integrity are supporting paths. The repository also documents bounded Jetstream2 pilot operations, experiment preparation and execution, and an agentic issue-to-review workflow. Relationships below are limited to README statements and SOURCE INDEX references; unsampled implementation details are not inferred.
Main entry points: download_batch.py Batch downloader · flowdc_pilot_cli.py Pilot lifecycle · workflow.py Workflow coordinator · flowdc_vine.py Vine runtime
Components 26
Dataset acquisition
- Manifest partitioning
bin/SplitParquet.py - Batch downloader
bin/download_batch.py - HTTP acquisition
bin/single_download.py - PAARC control
bin/download_batch.py - Output integrity
bin/flowdc_integrity.py
Distributed execution
- TaskVine manager
bin/TaskvineFLOWDC.py - Cloud manager
bin/TaskvineFLOWDCCloud.py - Vine runtime
bin/flowdc_vine.py - Vine worker
bin/flowdc_vine_worker.py - Vine protocol
bin/flowdc_vine_protocol.py
Pilot operations
- Operations CLI
bin/flowdc_ops.py - VM topology
bin/flowdc_topology.py - Pilot lifecycle
bin/flowdc_pilot_cli.py - Pilot journal
bin/flowdc_pilot_journal.py - Cloud provider
bin/flowdc_pilot_provider.py - Shutdown supervisor
bin/flowdc_pilot_supervisor.py
Research experiments
- Experiment runner
bin/flowdc_experiment.py - Experiment data
bin/flowdc_experiment_data.py - Research bridge
bin/flowdc_experiment_research.py
Agentic workflow
- Workflow coordinator
scripts/agentic/workflow.py - Task registry
scripts/agentic/tasks.py - Managed sessions
scripts/agentic/sessions.py - Review pipeline
scripts/agentic/pipeline.py - Independent review
scripts/agentic/review.py - Coverage gate
scripts/agentic/review_coverage.py
Other components
- Researcher
Connections 26
- Researcher → Manifest partitioning: partitions manifest
- Researcher → Batch downloader: starts download
- Batch downloader → PAARC control: controls concurrency
bin/download_batch.py - Batch downloader → HTTP acquisition: performs requests
bin/download_batch.py - Batch downloader → Output integrity: uses integrity
bin/download_batch.py - Researcher → TaskVine manager: starts distributed run
- TaskVine manager → Vine runtime: uses runtime
- Vine runtime → Vine worker: dispatches tasks
bin/flowdc_vine.py - Vine worker → Batch downloader: runs downloads
bin/flowdc_vine_worker.py - Vine runtime → Vine protocol: uses protocol
bin/flowdc_vine.py - Researcher → Cloud manager: starts cloud run
- Researcher → Operations CLI: runs operations
- Operations CLI → VM topology: uses topology
bin/flowdc_ops.py - Operations CLI → Pilot lifecycle: dispatches pilot
bin/flowdc_ops.py - Pilot lifecycle → Pilot journal: records state
bin/flowdc_pilot_cli.py - Pilot lifecycle → Cloud provider: uses provider
bin/flowdc_pilot_cli.py - Pilot lifecycle → Shutdown supervisor: controls supervisor
bin/flowdc_pilot_cli.py - Researcher → Experiment runner: runs experiment
- Experiment runner → Experiment data: prepares inputs
bin/flowdc_experiment.py - Experiment runner → Research bridge: uses research bridge
bin/flowdc_experiment.py - Researcher → Workflow coordinator: starts task workflow
- Workflow coordinator → Task registry: registers tasks
scripts/agentic/workflow.py - Workflow coordinator → Managed sessions: manages execution
scripts/agentic/workflow.py - Workflow coordinator → Review pipeline: coordinates review
scripts/agentic/workflow.py - Review pipeline → Independent review: invokes review
scripts/agentic/pipeline.py - Independent review → Coverage gate: checks coverage
scripts/agentic/review.py