# d3b-center/image-deid-etl architecture

> Architecture diagram of the GitHub repository d3b-center/image-deid-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/d3b-center/image-deid-etl
- Repository: https://github.com/d3b-center/image-deid-etl

## Overview

This is a command-line ETL for discovering Orthanc studies, mapping their DICOM metadata to program-specific subject/session labels, converting and organizing anonymized NIfTI outputs, and delivering results to Flywheel or S3. The main workflow is driven by the CLI; processed-study UUIDs are tracked in a database, while validation and image-review outputs support quality control. The graph reflects the README’s command capabilities and sampled runtime code; some command wiring and unsampled implementation details are therefore left at the documented level.

Main entry points: [`src/image_deid_etl/image_deid_etl/__main__.py`](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/__main__.py) (ETL commands), [`src/image_deid_etl/image_deid_etl/main_pipeline.py`](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/main_pipeline.py) (De-ID workflow), [`src/image_deid_etl/image_deid_etl/external_data_handling.py`](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/external_data_handling.py) (Subject mapping), [`src/image_deid_etl/image_deid_etl/custom_etl.py`](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_etl.py) (ETL transformations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI orchestration"]
  node_cli["ETL commands<br/>[__main__.py]"]
end

subgraph group_process["Study processing"]
  node_dicom["DICOM inspection<br/>[custom_dicom.py]"]
  node_pipeline["De-ID workflow<br/>[main_pipeline.py]"]
  node_external_mapping["Subject mapping"]
  node_etl_transform["ETL transformations<br/>[custom_etl.py]"]
end

subgraph group_outputs["Delivery and review"]
  node_nifti["NIfTI outputs"]
  node_review_images["Review images<br/>[images_no_save.py]"]
  node_flywheel_module["Flywheel operations<br/>[custom_flywheel.py]"]
end

subgraph group_state["State and sources"]
  node_orthanc["Orthanc client<br/>[orthanc.py]"]
  node_uuid_db[("Processed UUIDs<br/>[database.py]")]
  node_mapping_data["Mapping data"]
end

node_user(("CLI user"))
node_orthanc_api["Orthanc"]
node_flywheel_service["Flywheel"]
node_s3["S3 backup"]

node_user -->|"runs commands"| node_cli
node_cli -->|"checks studies"| node_orthanc
node_orthanc -->|"queries and downloads"| node_orthanc_api
node_cli -->|"reads or records UUIDs"| node_uuid_db
node_cli -->|"runs de-identification"| node_pipeline
node_pipeline -->|"resolves subject labels"| node_external_mapping
node_external_mapping -->|"reads mappings"| node_mapping_data
node_external_mapping -->|"uses DICOM metadata helpers"| node_etl_transform
node_pipeline -->|"converts and organizes"| node_etl_transform
node_etl_transform -->|"writes NIfTI files"| node_nifti
node_pipeline -->|"generates review images"| node_review_images
node_review_images -->|"reads NIfTI files"| node_nifti
node_cli -->|"uploads and adds metadata"| node_flywheel_module
node_flywheel_module -->|"uploads files and metadata"| node_flywheel_service
node_cli -->|"backs up NIfTIs"| node_s3

click node_cli "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/__main__.py"
click node_orthanc "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/orthanc.py"
click node_uuid_db "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/database.py"
click node_dicom "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_dicom.py"
click node_pipeline "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/main_pipeline.py"
click node_external_mapping "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/external_data_handling.py"
click node_etl_transform "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_etl.py"
click node_review_images "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/images_no_save.py"
click node_flywheel_module "https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_flywheel.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_user toneBlue
class node_dicom,node_pipeline,node_external_mapping,node_etl_transform toneAmber
class node_nifti,node_review_images,node_flywheel_module toneMint
class node_orthanc,node_uuid_db,node_mapping_data toneRose
class node_orthanc_api,node_flywheel_service,node_s3 toneIndigo
```

## Components

### CLI orchestration

- [ETL commands](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/__main__.py): `src/image_deid_etl/image_deid_etl/__main__.py`

### Study processing

- [DICOM inspection](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_dicom.py): `src/image_deid_etl/image_deid_etl/custom_dicom.py`
- [De-ID workflow](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/main_pipeline.py): `src/image_deid_etl/image_deid_etl/main_pipeline.py`
- [Subject mapping](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/external_data_handling.py): `src/image_deid_etl/image_deid_etl/external_data_handling.py`
- [ETL transformations](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_etl.py): `src/image_deid_etl/image_deid_etl/custom_etl.py`

### Delivery and review

- **NIfTI outputs**
- [Review images](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/images_no_save.py): `src/image_deid_etl/image_deid_etl/images_no_save.py`
- [Flywheel operations](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/custom_flywheel.py): `src/image_deid_etl/image_deid_etl/custom_flywheel.py`

### State and sources

- [Orthanc client](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/orthanc.py): `src/image_deid_etl/image_deid_etl/orthanc.py`
- [Processed UUIDs](https://github.com/d3b-center/image-deid-etl/blob/master/src/image_deid_etl/image_deid_etl/database.py): `src/image_deid_etl/image_deid_etl/database.py`
- **Mapping data**

### Other components

- **CLI user**
- **Orthanc**
- **Flywheel**
- **S3 backup**

## Connections

- CLI user → ETL commands: runs commands
- ETL commands → Orthanc client: checks studies
- Orthanc client → Orthanc: queries and downloads
- ETL commands → Processed UUIDs: reads or records UUIDs
- ETL commands → De-ID workflow: runs de-identification
- De-ID workflow → Subject mapping: resolves subject labels
- Subject mapping → Mapping data: reads mappings
- Subject mapping → ETL transformations: uses DICOM metadata helpers
- De-ID workflow → ETL transformations: converts and organizes
- ETL transformations → NIfTI outputs: writes NIfTI files
- De-ID workflow → Review images: generates review images
- Review images → NIfTI outputs: reads NIfTI files
- ETL commands → Flywheel operations: uploads and adds metadata
- Flywheel operations → Flywheel: uploads files and metadata
- ETL commands → S3 backup: backs up NIfTIs