# d3b-center/flywheel-soft-data-copy architecture

> Architecture diagram of the GitHub repository d3b-center/flywheel-soft-data-copy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/d3b-center/flywheel-soft-data-copy
- Repository: https://github.com/d3b-center/flywheel-soft-data-copy

## Overview

This repository is a collection of Flywheel data-management scripts. Its principal workflow loads CBTN subject IDs, discovers matching Flywheel project/subject/session records from either the D3b warehouse or Flywheel, writes a catalog CSV, then uses that catalog to soft-copy subjects or sessions into a destination project. Separate scripts queue Flywheel’s soft-copy gear and export a project file manifest. Source excerpts show the direct calls and data handling; external service behavior is represented only at the integration boundary.

Main entry points: [`find_fw_data.py`](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/find_fw_data.py) (Discovery CLI), [`fw_soft_copy_gear.py`](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_gear.py) (Gear runner), [`fw_soft_copy_sdk.py`](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py) (SDK copy CLI), [`utils.py`](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/utils.py) (Match finder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_discovery["Data discovery"]
  node_subject_sources["Subject inputs"]
  node_discover_cli["Discovery CLI<br/>[find_fw_data.py]"]
  node_subject_loader["Subject loader<br/>[find_fw_data.py]"]
  node_discovery_engine["Match finder<br/>[utils.py]"]
  node_warehouse[("D3b warehouse")]
  node_catalog["Match catalog CSV"]
end

subgraph group_copy["Soft copy"]
  node_sdk_cli["SDK copy CLI"]
  node_copy_planner["Copy planner"]
  node_soft_copy_api["Soft-copy operations"]
  node_gear_runner["Gear runner"]
end

subgraph group_manifest["Manifest export"]
  node_manifest_export["Manifest exporter"]
  node_project_view["Project DataView"]
  node_manifest_csv["Manifest CSV"]
end

node_user(("Operator"))
node_flywheel{{"Flywheel service"}}
node_soft_copy_gear{{"Soft-copy gear"}}

node_user -->|"runs discovery"| node_discover_cli
node_subject_sources -->|"provides IDs"| node_subject_loader
node_discover_cli -->|"loads subjects"| node_subject_loader
node_discover_cli -->|"calls finder"| node_discovery_engine
node_discovery_engine -->|"queries records"| node_warehouse
node_discovery_engine -->|"searches subjects"| node_flywheel
node_discover_cli -->|"authenticates"| node_flywheel
node_discover_cli -->|"writes matches"| node_catalog
node_catalog -->|"supplies rows"| node_sdk_cli
node_user -->|"runs copy"| node_sdk_cli
node_sdk_cli -->|"reads destination"| node_flywheel
node_sdk_cli -->|"checks and plans"| node_copy_planner
node_copy_planner -->|"looks up source"| node_flywheel
node_copy_planner -->|"dispatches copies"| node_soft_copy_api
node_soft_copy_api -->|"creates soft copies"| node_flywheel
node_user -->|"runs gear script"| node_gear_runner
node_gear_runner -->|"looks up projects"| node_flywheel
node_gear_runner -->|"queues jobs"| node_soft_copy_gear
node_gear_runner -->|"creates destination"| node_flywheel
node_user -->|"runs export"| node_manifest_export
node_manifest_export -->|"reads project view"| node_flywheel
node_flywheel -->|"provides rows"| node_project_view
node_manifest_export -->|"writes manifest"| node_manifest_csv

click node_discover_cli "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/find_fw_data.py"
click node_subject_loader "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/find_fw_data.py"
click node_discovery_engine "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/utils.py"
click node_sdk_cli "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py"
click node_copy_planner "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py"
click node_soft_copy_api "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py"
click node_gear_runner "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_gear.py"
click node_manifest_export "https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/pull_az_file_manifest.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_subject_sources,node_discover_cli,node_subject_loader,node_discovery_engine,node_warehouse,node_catalog toneBlue
class node_sdk_cli,node_copy_planner,node_soft_copy_api,node_gear_runner toneAmber
class node_manifest_export,node_project_view,node_manifest_csv toneMint
class node_user,node_flywheel,node_soft_copy_gear toneIndigo
```

## Components

### Data discovery

- **Subject inputs**
- [Discovery CLI](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/find_fw_data.py): `find_fw_data.py`
- [Subject loader](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/find_fw_data.py): `find_fw_data.py`
- [Match finder](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/utils.py): `utils.py`
- **D3b warehouse**
- **Match catalog CSV**

### Soft copy

- [SDK copy CLI](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py): `fw_soft_copy_sdk.py`
- [Copy planner](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py): `fw_soft_copy_sdk.py`
- [Soft-copy operations](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_sdk.py): `fw_soft_copy_sdk.py`
- [Gear runner](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/fw_soft_copy_gear.py): `fw_soft_copy_gear.py`

### Manifest export

- [Manifest exporter](https://github.com/d3b-center/flywheel-soft-data-copy/blob/main/pull_az_file_manifest.py): `pull_az_file_manifest.py`
- **Project DataView**
- **Manifest CSV**

### Other components

- **Operator**
- **Flywheel service**
- **Soft-copy gear**

## Connections

- Operator → Discovery CLI: runs discovery (evidence: `README.md`)
- Subject inputs → Subject loader: provides IDs (evidence: `find_fw_data.py`)
- Discovery CLI → Subject loader: loads subjects (evidence: `find_fw_data.py`)
- Discovery CLI → Match finder: calls finder (evidence: `find_fw_data.py`)
- Match finder → D3b warehouse: queries records (evidence: `utils.py`)
- Match finder → Flywheel service: searches subjects (evidence: `utils.py`)
- Discovery CLI → Flywheel service: authenticates (evidence: `find_fw_data.py`)
- Discovery CLI → Match catalog CSV: writes matches (evidence: `find_fw_data.py`)
- Match catalog CSV → SDK copy CLI: supplies rows (evidence: `README.md`)
- Operator → SDK copy CLI: runs copy (evidence: `README.md`)
- SDK copy CLI → Flywheel service: reads destination (evidence: `fw_soft_copy_sdk.py`)
- SDK copy CLI → Copy planner: checks and plans (evidence: `fw_soft_copy_sdk.py`)
- Copy planner → Flywheel service: looks up source (evidence: `fw_soft_copy_sdk.py`)
- Copy planner → Soft-copy operations: dispatches copies (evidence: `fw_soft_copy_sdk.py`)
- Soft-copy operations → Flywheel service: creates soft copies (evidence: `fw_soft_copy_sdk.py`)
- Operator → Gear runner: runs gear script (evidence: `fw_soft_copy_gear.py`)
- Gear runner → Flywheel service: looks up projects (evidence: `fw_soft_copy_gear.py`)
- Gear runner → Soft-copy gear: queues jobs (evidence: `fw_soft_copy_gear.py`)
- Gear runner → Flywheel service: creates destination (evidence: `fw_soft_copy_gear.py`)
- Operator → Manifest exporter: runs export (evidence: `README.md`)
- Manifest exporter → Flywheel service: reads project view (evidence: `pull_az_file_manifest.py`)
- Flywheel service → Project DataView: provides rows
- Manifest exporter → Manifest CSV: writes manifest (evidence: `pull_az_file_manifest.py`)