# erikosinga/meerkatpolpipeline architecture

> Architecture diagram of the GitHub repository erikosinga/meerkatpolpipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/erikosinga/meerkatpolpipeline
- Repository: https://github.com/erikosinga/meerkatpolpipeline

## Overview

#### Purpose and execution
`meerkatpolpipeline` is a Python 3.11+ orchestration pipeline for MeerKAT polarisation reduction: raw-data acquisition through calibration, imaging, RM synthesis, validation, and early science products. Its primary executable is `meerkatpolpipeline/flows/Lband_reduction.py`, invoked with a YAML configuration and working directory. Configuration loading and runtime option handling are centralized in `meerkatpolpipeline/configuration.py` and `meerkatpolpipeline/options.py`.

#### Workflow orchestration
The L-band flow composes processing stages as Prefect tasks/flows, giving dependency tracking, logs, and run visibility through a local or remote Prefect API/server. The workflow boundary is Python orchestration versus externally executed radio-astronomy software. Key paths:
- `meerkatpolpipeline/flows/Lband_reduction.py`
- `meerkatpolpipeline/logging.py`
- `templates/pipeline-template-config.yaml`

#### Data acquisition and Measurement Set preparation
The pipeline begins by downloading raw observation products, applying data clipping and parallactic-angle correction, and operating on radio-interferometric Measurement Sets (MSs). MS-specific operations are encapsulated separately from flow construction:
- `meerkatpolpipeline/download/download.py`
- `meerkatpolpipeline/download/correct_parang.py`
- `meerkatpolpipeline/measurementset.py`

#### Cross-calibration through CARACal and CASA
Direction-independent polarisation calibration is delegated to CARACal, with pipeline adapters creating/running CARACal strategies and CASA scripts. CASA-based cross-calibration includes calibrator-specific logic and Stokes-Q/U/V defaults. CARACal is an externally installed dependency, commonly containerized.
- `meerkatpolpipeline/caracal/_caracal.py`
- `meerkatpolpipeline/casacrosscal/casacrosscal.py`
- `meerkatpolpipeline/casacrosscal/casa_script_crosscal.py`

#### Calibration quality gates and self-calibration
Post-calibration checks compare results to calibrator models; optional self-calibration supports direction-independent/direction-dependent refinement. The latter relies on a Singularity container carrying LOFAR software, establishing a substantial external-runtime boundary.
- `meerkatpolpipeline/check_calibrator/check_calibrator.py`
- `meerkatpolpipeline/selfcal/_facetselfcal.py`
- `meerkatpolpipeline/caracal/field_intents.py`

#### Imaging and polarisation cube construction
Imaging is driven through WSClean, then channel images are beam-harmonized, combined into cubes, and primary-beam corrected. This stage transforms calibrated MS visibility data into frequency-resolved Stokes image products.
- `meerkatpolpipeline/wsclean/wsclean.py`
- `meerkatpolpipeline/cube_imaging/cube_imaging.py`
- `meerkatpolpipeline/cube_imaging/combine_to_imagecube.py`

#### Faraday analysis, validation, and science outputs
Image cubes feed 1D/3D RM synthesis and validation. Supporting checks quantify flagging and image noise by frequency, while external survey comparisons assess target consistency against NVSS and RACS. Preliminary science plotting consumes RM-synthesis outputs.
- `meerkatpolpipeline/rmsynth/rmsynth1d.py`
- `meerkatpolpipeline/rmsynth/rmsynth3d.py`
- `meerkatpolpipeline/validation/validate_field.py`
- `meerkatpolpipeline/check_nvss/target_vs_nvss.py`
- `meerkatpolpipeline/check_racs/target_vs_racs.py`

#### Runtime and operational constraints
Python coordinates work, while CARACal, CASA, WSClean, and optional LOFAR/Singularity tooling perform domain processing. Prefect’s API endpoint is the operational observability service; it can be local or reached through an SSH tunnel. Large cube workloads require high file-descriptor limits; noninteractive plotting should use the Agg Matplotlib backend.

Main entry points: [`meerkatpolpipeline/flows/Lband_reduction.py`](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/flows/Lband_reduction.py) (L-band reduction flow), [`meerkatpolpipeline/measurementset.py`](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/measurementset.py) (MS preparation), [`meerkatpolpipeline/check_calibrator/check_calibrator.py`](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/check_calibrator/check_calibrator.py) (Calibrator quality gate), [`meerkatpolpipeline/wsclean/wsclean.py`](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/wsclean/wsclean.py) (WSClean imaging)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_config["YAML configuration<br/>pipeline config"]
  node_options["Runtime options<br/>Python options<br/>[options.py]"]
  node_lband_flow{{"L-band reduction flow<br/>Prefect flow<br/>[Lband_reduction.py]"}}
  node_logging["Pipeline logging<br/>Python logging<br/>[logging.py]"]
  node_prefect_api[("Prefect API/server<br/>observability service")]
end

subgraph group_preparation["Data preparation"]
  node_download["Raw data download<br/>acquisition task<br/>[download.py]"]
  node_ms_prep["MS preparation<br/>Measurement Set operations<br/>[measurementset.py]"]
  node_parang["Parallactic correction<br/>MS correction<br/>[correct_parang.py]"]
end

subgraph group_calibration["Calibration"]
  node_caracal{{"CARACal calibration<br/>external calibration adapter<br/>[_caracal.py]"}}
  node_casa_crosscal["CASA cross-calibration<br/>calibration task<br/>[casacrosscal.py]"]
  node_cal_quality["Calibrator quality gate<br/>validation task"]
  node_selfcal["Optional self-calibration<br/>refinement task<br/>[_facetselfcal.py]"]
end

subgraph group_imaging["Imaging"]
  node_wsclean{{"WSClean imaging<br/>external imaging adapter<br/>[wsclean.py]"}}
  node_cube_imaging["Polarisation cube build<br/>image-cube processing<br/>[cube_imaging.py]"]
end

subgraph group_analysis["Faraday analysis"]
  node_rm_synthesis["RM synthesis<br/>Faraday analysis<br/>[rmsynth1d.py]"]
  node_field_validation["Field validation<br/>quality analysis<br/>[validate_field.py]"]
  node_science_outputs["Early science products<br/>plots and products"]
end

node_radio_software{{"Radio software runtimes<br/>external runtime"}}

node_config -->|"configures"| node_lband_flow
node_options -->|"sets runtime behavior"| node_lband_flow
node_lband_flow -->|"emits logs"| node_logging
node_lband_flow -->|"reports runs"| node_prefect_api
node_lband_flow -->|"orchestrates"| node_download
node_download -->|"raw MS products"| node_ms_prep
node_parang -->|"corrects"| node_ms_prep
node_ms_prep -->|"prepared MSs"| node_caracal
node_ms_prep -->|"prepared MSs"| node_casa_crosscal
node_caracal -->|"calibrated data"| node_cal_quality
node_casa_crosscal -->|"calibrated data"| node_cal_quality
node_cal_quality -.->|"optional refinement"| node_selfcal
node_cal_quality -->|"accepted calibrated MSs"| node_wsclean
node_selfcal -.->|"refined MSs"| node_wsclean
node_wsclean -->|"channel images"| node_cube_imaging
node_cube_imaging -->|"Stokes image cubes"| node_rm_synthesis
node_cube_imaging -->|"image quality inputs"| node_field_validation
node_rm_synthesis -->|"Faraday products"| node_science_outputs
node_field_validation -->|"validated results"| node_science_outputs
node_caracal -->|"executes"| node_radio_software
node_casa_crosscal -->|"executes CASA"| node_radio_software
node_selfcal -.->|"uses optional LOFAR runtime"| node_radio_software
node_wsclean -->|"executes WSClean"| node_radio_software

click node_config "https://github.com/erikosinga/meerkatpolpipeline/blob/main/templates/pipeline-template-config.yaml"
click node_options "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/options.py"
click node_lband_flow "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/flows/Lband_reduction.py"
click node_logging "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/logging.py"
click node_download "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/download/download.py"
click node_ms_prep "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/measurementset.py"
click node_parang "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/download/correct_parang.py"
click node_caracal "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/caracal/_caracal.py"
click node_casa_crosscal "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/casacrosscal/casacrosscal.py"
click node_cal_quality "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/check_calibrator/check_calibrator.py"
click node_selfcal "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/selfcal/_facetselfcal.py"
click node_wsclean "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/wsclean/wsclean.py"
click node_cube_imaging "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/cube_imaging/cube_imaging.py"
click node_rm_synthesis "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/rmsynth/rmsynth1d.py"
click node_field_validation "https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/validation/validate_field.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_options,node_lband_flow,node_logging,node_prefect_api toneBlue
class node_download,node_ms_prep,node_parang toneAmber
class node_caracal,node_casa_crosscal,node_cal_quality,node_selfcal toneMint
class node_wsclean,node_cube_imaging toneRose
class node_rm_synthesis,node_field_validation,node_science_outputs toneIndigo
class node_radio_software toneNeutral
```

## Components

### Orchestration

- [YAML configuration](https://github.com/erikosinga/meerkatpolpipeline/blob/main/templates/pipeline-template-config.yaml): pipeline config · `templates/pipeline-template-config.yaml`
- [Runtime options](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/options.py): Python options · `meerkatpolpipeline/options.py`
- [L-band reduction flow](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/flows/Lband_reduction.py): Prefect flow · Primary pipeline executable. · `meerkatpolpipeline/flows/Lband_reduction.py`
- [Pipeline logging](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/logging.py): Python logging · `meerkatpolpipeline/logging.py`
- **Prefect API/server**: observability service · Local or remote run tracking endpoint.

### Data preparation

- [Raw data download](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/download/download.py): acquisition task · `meerkatpolpipeline/download/download.py`
- [MS preparation](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/measurementset.py): Measurement Set operations · Clipping and parallactic-angle correction for Measurement Sets. · `meerkatpolpipeline/measurementset.py`
- [Parallactic correction](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/download/correct_parang.py): MS correction · `meerkatpolpipeline/download/correct_parang.py`

### Calibration

- [CARACal calibration](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/caracal/_caracal.py): external calibration adapter · Builds and runs CARACal strategies. · `meerkatpolpipeline/caracal/_caracal.py`
- [CASA cross-calibration](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/casacrosscal/casacrosscal.py): calibration task · Calibrator-specific Stokes Q/U/V calibration. · `meerkatpolpipeline/casacrosscal/casacrosscal.py`
- [Calibrator quality gate](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/check_calibrator/check_calibrator.py): validation task · `meerkatpolpipeline/check_calibrator/check_calibrator.py`
- [Optional self-calibration](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/selfcal/_facetselfcal.py): refinement task · Supports DI and LOFAR/Singularity-backed DD refinement. · `meerkatpolpipeline/selfcal/_facetselfcal.py`

### Imaging

- [WSClean imaging](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/wsclean/wsclean.py): external imaging adapter · `meerkatpolpipeline/wsclean/wsclean.py`
- [Polarisation cube build](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/cube_imaging/cube_imaging.py): image-cube processing · Beam harmonizes, combines channels, and applies primary-beam correction. · `meerkatpolpipeline/cube_imaging/cube_imaging.py`

### Faraday analysis

- [RM synthesis](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/rmsynth/rmsynth1d.py): Faraday analysis · Runs 1D and 3D analysis on polarisation cubes. · `meerkatpolpipeline/rmsynth/rmsynth1d.py`
- [Field validation](https://github.com/erikosinga/meerkatpolpipeline/blob/main/meerkatpolpipeline/validation/validate_field.py): quality analysis · Checks flagging, noise, and survey consistency. · `meerkatpolpipeline/validation/validate_field.py`
- **Early science products**: plots and products · Consumes RM-synthesis outputs.

### Other components

- **Radio software runtimes**: external runtime · CARACal, CASA, WSClean, and optional LOFAR/Singularity tooling.

## Connections

- YAML configuration → L-band reduction flow: configures
- Runtime options → L-band reduction flow: sets runtime behavior
- L-band reduction flow → Pipeline logging: emits logs
- L-band reduction flow → Prefect API/server: reports runs
- L-band reduction flow → Raw data download: orchestrates
- Raw data download → MS preparation: raw MS products
- Parallactic correction → MS preparation: corrects
- MS preparation → CARACal calibration: prepared MSs
- MS preparation → CASA cross-calibration: prepared MSs
- CARACal calibration → Calibrator quality gate: calibrated data
- CASA cross-calibration → Calibrator quality gate: calibrated data
- Calibrator quality gate → Optional self-calibration: optional refinement
- Calibrator quality gate → WSClean imaging: accepted calibrated MSs
- Optional self-calibration → WSClean imaging: refined MSs
- WSClean imaging → Polarisation cube build: channel images
- Polarisation cube build → RM synthesis: Stokes image cubes
- Polarisation cube build → Field validation: image quality inputs
- RM synthesis → Early science products: Faraday products
- Field validation → Early science products: validated results
- CARACal calibration → Radio software runtimes: executes
- CASA cross-calibration → Radio software runtimes: executes CASA
- Optional self-calibration → Radio software runtimes: uses optional LOFAR runtime
- WSClean imaging → Radio software runtimes: executes WSClean