# mhardcastle/ddf-pipeline architecture

> Architecture diagram of the GitHub repository mhardcastle/ddf-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mhardcastle/ddf-pipeline
- Repository: https://github.com/mhardcastle/ddf-pipeline

## Overview

ddf-pipeline reduces LOFAR continuum Stokes-I data through direction-dependent self-calibration and imaging to science-ready products. The main runtime is the iterative reduction in `scripts/pipeline.py`; field-level reprocessing stages data and invokes that reduction, with additional product-generation and quality-analysis branches. The README establishes the product goal, while source-index references support the shown module relationships; unsampled capabilities are represented without asserting unverified call paths.

Main entry points: [`scripts/pipeline.py`](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/pipeline.py) (Self-calibration pipeline), [`scripts/quality_pipeline.py`](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/quality_pipeline.py) (Image quality checks), [`utils/auxcodes.py`](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/auxcodes.py) (Imaging utilities), [`scripts/run_full_field_reprocessing_pipeline.py`](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/run_full_field_reprocessing_pipeline.py) (Field reprocessing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Field orchestration"]
  node_field_reprocessing["Field reprocessing"]
  node_reprocessing_support["Staging and transfer"]
end

subgraph group_reduction["Calibration and imaging"]
  node_main_pipeline["Self-calibration pipeline<br/>[pipeline.py]"]
  node_measurement_lists["Measurement-set lists<br/>[make_mslists.py]"]
end

subgraph group_products["Science products"]
  node_polarization_cubes["Polarization cubes<br/>[do_polcubes.py]"]
  node_spectral_products["Spectral products"]
end

subgraph group_quality["Quality analysis"]
  node_quality_pipeline["Image quality checks"]
  node_quality_plots["Quality plots"]
  node_crossmatch["Catalogue crossmatch"]
  node_facet_offsets["Facet offsets<br/>[facet_offsets.py]"]
  node_dynamic_range["Dynamic range check<br/>[dr_checker.py]"]
  node_source_counts["Source-count fitting"]
end

subgraph group_shared["Shared utilities"]
  node_options["Option parsing<br/>[options.py]"]
  node_parameter_set["Pipeline parameters<br/>[parset.py]"]
  node_auxiliary_codes["Imaging utilities<br/>[auxcodes.py]"]
  node_download["File download<br/>[download_file.py]"]
  node_remote_copy["Remote storage access<br/>[rclone.py]"]
  node_cpu_detection["CPU detection<br/>[getcpus.py]"]
  node_reprocessing_helpers["Reprocessing helpers"]
end

node_operator(("Data operator"))
node_imaging_tool["DDFacet and KillMS"]
node_remote_storage[("Remote storage")]

node_operator -.->|"starts reprocessing"| node_field_reprocessing
node_operator -.->|"runs reduction"| node_main_pipeline
node_field_reprocessing -->|"runs reduction"| node_main_pipeline
node_field_reprocessing -->|"generates cubes"| node_polarization_cubes
node_field_reprocessing -->|"stages data"| node_reprocessing_support
node_reprocessing_support -->|"uses transfer"| node_remote_copy
node_reprocessing_support -->|"downloads files"| node_download
node_reprocessing_support -.->|"stages from"| node_remote_storage
node_main_pipeline -.->|"runs imaging"| node_imaging_tool
node_main_pipeline -->|"uses options"| node_options
node_main_pipeline -->|"uses parameters"| node_parameter_set
node_main_pipeline -->|"uses utilities"| node_auxiliary_codes
node_main_pipeline -.->|"uses lists"| node_measurement_lists
node_main_pipeline -->|"generates products"| node_spectral_products
node_main_pipeline -->|"generates cubes"| node_polarization_cubes
node_measurement_lists -->|"uses utilities"| node_auxiliary_codes
node_polarization_cubes -->|"uses pipeline"| node_main_pipeline
node_polarization_cubes -->|"uses utilities"| node_auxiliary_codes
node_spectral_products -->|"uses pipeline"| node_main_pipeline
node_spectral_products -->|"uses utilities"| node_auxiliary_codes
node_quality_pipeline -->|"uses options"| node_options
node_quality_pipeline -->|"uses utilities"| node_auxiliary_codes
node_quality_pipeline -->|"crossmatches catalogues"| node_crossmatch
node_quality_pipeline -->|"creates plots"| node_quality_plots
node_quality_pipeline -->|"checks offsets"| node_facet_offsets
node_quality_pipeline -->|"checks dynamic range"| node_dynamic_range
node_quality_pipeline -->|"fits source counts"| node_source_counts
node_quality_plots -->|"uses matching"| node_crossmatch
node_quality_plots -->|"uses utilities"| node_auxiliary_codes
node_dynamic_range -->|"uses matching"| node_crossmatch
node_source_counts -->|"uses utilities"| node_auxiliary_codes
node_parameter_set -->|"gets CPU count"| node_cpu_detection
node_options -->|"uses formatting"| node_auxiliary_codes
node_options -->|"reads parameter definitions"| node_parameter_set
node_reprocessing_helpers -->|"uses transfer"| node_remote_copy
node_reprocessing_helpers -->|"uses downloads"| node_download
node_remote_copy -->|"transfers files"| node_remote_storage

click node_field_reprocessing "https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/run_full_field_reprocessing_pipeline.py"
click node_reprocessing_support "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/reprocessing_utils.py"
click node_main_pipeline "https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/pipeline.py"
click node_measurement_lists "https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/make_mslists.py"
click node_polarization_cubes "https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/do_polcubes.py"
click node_spectral_products "https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/do_spectral_restored.py"
click node_quality_pipeline "https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/quality_pipeline.py"
click node_quality_plots "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/quality_make_plots.py"
click node_crossmatch "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/crossmatch_utils.py"
click node_facet_offsets "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/facet_offsets.py"
click node_dynamic_range "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/dr_checker.py"
click node_source_counts "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/fit_sourcecounts.py"
click node_options "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/options.py"
click node_parameter_set "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/parset.py"
click node_auxiliary_codes "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/auxcodes.py"
click node_download "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/download_file.py"
click node_remote_copy "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/rclone.py"
click node_cpu_detection "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/getcpus.py"
click node_reprocessing_helpers "https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/reprocessing_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_field_reprocessing,node_reprocessing_support toneBlue
class node_main_pipeline,node_measurement_lists,node_remote_storage toneAmber
class node_polarization_cubes,node_spectral_products toneMint
class node_quality_pipeline,node_quality_plots,node_crossmatch,node_facet_offsets,node_dynamic_range,node_source_counts toneRose
class node_options,node_parameter_set,node_auxiliary_codes,node_download,node_remote_copy,node_cpu_detection,node_reprocessing_helpers,node_operator,node_imaging_tool toneIndigo
```

## Components

### Field orchestration

- [Field reprocessing](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/run_full_field_reprocessing_pipeline.py): `scripts/run_full_field_reprocessing_pipeline.py`
- [Staging and transfer](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/reprocessing_utils.py): `utils/reprocessing_utils.py`

### Calibration and imaging

- [Self-calibration pipeline](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/pipeline.py): `scripts/pipeline.py`
- [Measurement-set lists](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/make_mslists.py): `scripts/make_mslists.py`

### Science products

- [Polarization cubes](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/do_polcubes.py): `scripts/do_polcubes.py`
- [Spectral products](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/do_spectral_restored.py): `scripts/do_spectral_restored.py`

### Quality analysis

- [Image quality checks](https://github.com/mhardcastle/ddf-pipeline/blob/master/scripts/quality_pipeline.py): `scripts/quality_pipeline.py`
- [Quality plots](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/quality_make_plots.py): `utils/quality_make_plots.py`
- [Catalogue crossmatch](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/crossmatch_utils.py): `utils/crossmatch_utils.py`
- [Facet offsets](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/facet_offsets.py): `utils/facet_offsets.py`
- [Dynamic range check](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/dr_checker.py): `utils/dr_checker.py`
- [Source-count fitting](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/fit_sourcecounts.py): `utils/fit_sourcecounts.py`

### Shared utilities

- [Option parsing](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/options.py): `utils/options.py`
- [Pipeline parameters](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/parset.py): `utils/parset.py`
- [Imaging utilities](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/auxcodes.py): `utils/auxcodes.py`
- [File download](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/download_file.py): `utils/download_file.py`
- [Remote storage access](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/rclone.py): `utils/rclone.py`
- [CPU detection](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/getcpus.py): `utils/getcpus.py`
- [Reprocessing helpers](https://github.com/mhardcastle/ddf-pipeline/blob/master/utils/reprocessing_utils.py): `utils/reprocessing_utils.py`

### Other components

- **Data operator**
- **DDFacet and KillMS**
- **Remote storage**

## Connections

- Data operator → Field reprocessing: starts reprocessing
- Data operator → Self-calibration pipeline: runs reduction
- Field reprocessing → Self-calibration pipeline: runs reduction (evidence: `scripts/run_full_field_reprocessing_pipeline.py`)
- Field reprocessing → Polarization cubes: generates cubes (evidence: `scripts/run_full_field_reprocessing_pipeline.py`)
- Field reprocessing → Staging and transfer: stages data (evidence: `scripts/run_full_field_reprocessing_pipeline.py`)
- Staging and transfer → Remote storage access: uses transfer (evidence: `utils/reprocessing_utils.py`)
- Staging and transfer → File download: downloads files (evidence: `utils/reprocessing_utils.py`)
- Staging and transfer → Remote storage: stages from
- Self-calibration pipeline → DDFacet and KillMS: runs imaging
- Self-calibration pipeline → Option parsing: uses options (evidence: `scripts/pipeline.py`)
- Self-calibration pipeline → Pipeline parameters: uses parameters (evidence: `scripts/pipeline.py`)
- Self-calibration pipeline → Imaging utilities: uses utilities (evidence: `scripts/pipeline.py`)
- Self-calibration pipeline → Measurement-set lists: uses lists
- Self-calibration pipeline → Spectral products: generates products (evidence: `scripts/pipeline.py`)
- Self-calibration pipeline → Polarization cubes: generates cubes (evidence: `scripts/pipeline.py`)
- Measurement-set lists → Imaging utilities: uses utilities (evidence: `scripts/make_mslists.py`)
- Polarization cubes → Self-calibration pipeline: uses pipeline (evidence: `scripts/do_polcubes.py`)
- Polarization cubes → Imaging utilities: uses utilities (evidence: `scripts/do_polcubes.py`)
- Spectral products → Self-calibration pipeline: uses pipeline (evidence: `scripts/do_spectral_restored.py`)
- Spectral products → Imaging utilities: uses utilities (evidence: `scripts/do_spectral_restored.py`)
- Image quality checks → Option parsing: uses options (evidence: `scripts/quality_pipeline.py`)
- Image quality checks → Imaging utilities: uses utilities (evidence: `scripts/quality_pipeline.py`)
- Image quality checks → Catalogue crossmatch: crossmatches catalogues (evidence: `scripts/quality_pipeline.py`)
- Image quality checks → Quality plots: creates plots (evidence: `scripts/quality_pipeline.py`)
- Image quality checks → Facet offsets: checks offsets (evidence: `scripts/quality_pipeline.py`)
- Image quality checks → Dynamic range check: checks dynamic range (evidence: `scripts/quality_pipeline.py`)
- Image quality checks → Source-count fitting: fits source counts (evidence: `scripts/quality_pipeline.py`)
- Quality plots → Catalogue crossmatch: uses matching (evidence: `utils/quality_make_plots.py`)
- Quality plots → Imaging utilities: uses utilities (evidence: `utils/quality_make_plots.py`)
- Dynamic range check → Catalogue crossmatch: uses matching (evidence: `utils/dr_checker.py`)
- Source-count fitting → Imaging utilities: uses utilities (evidence: `utils/fit_sourcecounts.py`)
- Pipeline parameters → CPU detection: gets CPU count (evidence: `utils/parset.py`)
- Option parsing → Imaging utilities: uses formatting (evidence: `utils/options.py`)
- Option parsing → Pipeline parameters: reads parameter definitions (evidence: `utils/options.py`)
- Reprocessing helpers → Remote storage access: uses transfer (evidence: `utils/reprocessing_utils.py`)
- Reprocessing helpers → File download: uses downloads (evidence: `utils/reprocessing_utils.py`)
- Remote storage access → Remote storage: transfers files (evidence: `utils/rclone.py`)
