erikosinga/meerkatpolpipeline

Purpose and execution

meerkatpolpipeline is a Python 3.11+ orchestration pipeline for MeerKAT polarisation reduction: raw-data acquisition through calibration, imaging, RM synthesis, validation, and early science products. Its primary executable is meerkatpolpipeline/flows/Lband_reduction.py, invoked with a YAML configuration and working directory. Configuration loading and runtime option handling are centralized in meerkatpolpipeline/configuration.py and meerkatpolpipeline/options.py.

Workflow orchestration

The L-band flow composes processing stages as Prefect tasks/flows, giving dependency tracking, logs, and run visibility through a local or remote Prefect API/server. The workflow boundary is Python orchestration versus externally executed radio-astronomy software. Key paths:

• meerkatpolpipeline/flows/Lband_reduction.py

• meerkatpolpipeline/logging.py

• templates/pipeline-template-config.yaml

Data acquisition and Measurement Set preparation

The pipeline begins by downloading raw observation products, applying data clipping and parallactic-angle correction, and operating on radio-interferometric Measurement Sets (MSs). MS-specific operations are encapsulated separately from flow construction:

• meerkatpolpipeline/download/download.py

• meerkatpolpipeline/download/correct_parang.py

• meerkatpolpipeline/measurementset.py

Cross-calibration through CARACal and CASA

Direction-independent polarisation calibration is delegated to CARACal, with pipeline adapters creating/running CARACal strategies and CASA scripts. CASA-based cross-calibration includes calibrator-specific logic and Stokes-Q/U/V defaults. CARACal is an externally installed dependency, commonly containerized.

• meerkatpolpipeline/caracal/_caracal.py

• meerkatpolpipeline/casacrosscal/casacrosscal.py

• meerkatpolpipeline/casacrosscal/casa_script_crosscal.py

Calibration quality gates and self-calibration

Post-calibration checks compare results to calibrator models; optional self-calibration supports direction-independent/direction-dependent refinement. The latter relies on a Singularity container carrying LOFAR software, establishing a substantial external-runtime boundary.

• meerkatpolpipeline/check_calibrator/check_calibrator.py

• meerkatpolpipeline/selfcal/_facetselfcal.py

• meerkatpolpipeline/caracal/field_intents.py

Imaging and polarisation cube construction

Imaging is driven through WSClean, then channel images are beam-harmonized, combined into cubes, and primary-beam corrected. This stage transforms calibrated MS visibility data into frequency-resolved Stokes image products.

• meerkatpolpipeline/wsclean/wsclean.py

• meerkatpolpipeline/cube_imaging/cube_imaging.py

• meerkatpolpipeline/cube_imaging/combine_to_imagecube.py

Faraday analysis, validation, and science outputs

Image cubes feed 1D/3D RM synthesis and validation. Supporting checks quantify flagging and image noise by frequency, while external survey comparisons assess target consistency against NVSS and RACS. Preliminary science plotting consumes RM-synthesis outputs.

• meerkatpolpipeline/rmsynth/rmsynth1d.py

• meerkatpolpipeline/rmsynth/rmsynth3d.py

• meerkatpolpipeline/validation/validate_field.py

• meerkatpolpipeline/check_nvss/target_vs_nvss.py

• meerkatpolpipeline/check_racs/target_vs_racs.py

Runtime and operational constraints

Python coordinates work, while CARACal, CASA, WSClean, and optional LOFAR/Singularity tooling perform domain processing. Prefect’s API endpoint is the operational observability service; it can be local or reached through an SSH tunnel. Large cube workloads require high file-descriptor limits; noninteractive plotting should use the Agg Matplotlib backend.