How mhardcastle/ddf-pipeline works
ddf-pipeline reduces LOFAR continuum Stokes-I data through direction-dependent self-calibration and imaging to science-ready products. The main runtime is the iterative reduction in scripts/pipeline.py; field-level reprocessing stages data and invokes that reduction, with additional product-generation and quality-analysis branches. The README establishes the product goal, while source-index references support the shown module relationships; unsampled capabilities are represented without asserting unverified call paths.
Main entry points: pipeline.py Self-calibration pipeline · quality_pipeline.py Image quality checks · auxcodes.py Imaging utilities · run_full_field_reprocessing_pipeline.py Field reprocessing
Components 22
Field orchestration
- Field reprocessing
scripts/run_full_field_reprocessing_pipeline.py - Staging and transfer
utils/reprocessing_utils.py
Calibration and imaging
- Self-calibration pipeline
scripts/pipeline.py - Measurement-set lists
scripts/make_mslists.py
Science products
- Polarization cubes
scripts/do_polcubes.py - Spectral products
scripts/do_spectral_restored.py
Quality analysis
- Image quality checks
scripts/quality_pipeline.py - Quality plots
utils/quality_make_plots.py - Catalogue crossmatch
utils/crossmatch_utils.py - Facet offsets
utils/facet_offsets.py - Dynamic range check
utils/dr_checker.py - Source-count fitting
utils/fit_sourcecounts.py
Shared utilities
- Option parsing
utils/options.py - Pipeline parameters
utils/parset.py - Imaging utilities
utils/auxcodes.py - File download
utils/download_file.py - Remote storage access
utils/rclone.py - CPU detection
utils/getcpus.py - Reprocessing helpers
utils/reprocessing_utils.py
Other components
- Data operator
- DDFacet and KillMS
- Remote storage
Connections 37
- Data operator → Field reprocessing: starts reprocessing
- Data operator → Self-calibration pipeline: runs reduction
- Field reprocessing → Self-calibration pipeline: runs reduction
scripts/run_full_field_reprocessing_pipeline.py - Field reprocessing → Polarization cubes: generates cubes
scripts/run_full_field_reprocessing_pipeline.py - Field reprocessing → Staging and transfer: stages data
scripts/run_full_field_reprocessing_pipeline.py - Staging and transfer → Remote storage access: uses transfer
utils/reprocessing_utils.py - Staging and transfer → File download: downloads files
utils/reprocessing_utils.py - Staging and transfer → Remote storage: stages from
- Self-calibration pipeline → DDFacet and KillMS: runs imaging
- Self-calibration pipeline → Option parsing: uses options
scripts/pipeline.py - Self-calibration pipeline → Pipeline parameters: uses parameters
scripts/pipeline.py - Self-calibration pipeline → Imaging utilities: uses utilities
scripts/pipeline.py - Self-calibration pipeline → Measurement-set lists: uses lists
- Self-calibration pipeline → Spectral products: generates products
scripts/pipeline.py - Self-calibration pipeline → Polarization cubes: generates cubes
scripts/pipeline.py - Measurement-set lists → Imaging utilities: uses utilities
scripts/make_mslists.py - Polarization cubes → Self-calibration pipeline: uses pipeline
scripts/do_polcubes.py - Polarization cubes → Imaging utilities: uses utilities
scripts/do_polcubes.py - Spectral products → Self-calibration pipeline: uses pipeline
scripts/do_spectral_restored.py - Spectral products → Imaging utilities: uses utilities
scripts/do_spectral_restored.py - Image quality checks → Option parsing: uses options
scripts/quality_pipeline.py - Image quality checks → Imaging utilities: uses utilities
scripts/quality_pipeline.py - Image quality checks → Catalogue crossmatch: crossmatches catalogues
scripts/quality_pipeline.py - Image quality checks → Quality plots: creates plots
scripts/quality_pipeline.py - Image quality checks → Facet offsets: checks offsets
scripts/quality_pipeline.py - Image quality checks → Dynamic range check: checks dynamic range
scripts/quality_pipeline.py - Image quality checks → Source-count fitting: fits source counts
scripts/quality_pipeline.py - Quality plots → Catalogue crossmatch: uses matching
utils/quality_make_plots.py - Quality plots → Imaging utilities: uses utilities
utils/quality_make_plots.py - Dynamic range check → Catalogue crossmatch: uses matching
utils/dr_checker.py - Source-count fitting → Imaging utilities: uses utilities
utils/fit_sourcecounts.py - Pipeline parameters → CPU detection: gets CPU count
utils/parset.py - Option parsing → Imaging utilities: uses formatting
utils/options.py - Option parsing → Pipeline parameters: reads parameter definitions
utils/options.py - Reprocessing helpers → Remote storage access: uses transfer
utils/reprocessing_utils.py - Reprocessing helpers → File download: uses downloads
utils/reprocessing_utils.py - Remote storage access → Remote storage: transfers files
utils/rclone.py