How kpenev/autowisp works
1. Overall shape
- AutoWISP is a Python-based astronomical photometry pipeline with two major faces:
- a core processing library under autowisp/
- a Django browser interface under autowisp/browser_interface/
- The repository is organized around a workflow that turns raw FITS images into calibrated images, extracted measurements, fitted magnitudes, light curves, and finally detrended science products.
2. Primary data flow
- The main pipeline path is: raw FITS input → calibration/master-frame generation → astrometry/source matching → photometry/PSF-PRF fitting → ensemble magnitude fitting → light-curve assembly → EPD/TFA detrending.
- Each stage writes artifacts that downstream stages consume, so the system is not a single monolithic run; it is a sequence of persisted processing steps.
3. Core image-processing subsystem
- autowisp/image_calibration/, autowisp/astrometry/, autowisp/source_finder.py, autowisp/fit_expression/, autowisp/magnitude_fitting/, and autowisp/light_curves/ implement the scientific algorithms.
- These modules handle the mathematically intensive parts: overscan correction, master bias/dark/flat creation, coordinate transformations, PSF/PRF fitting, aperture photometry, regression-based magnitude fitting, and light-curve corrections.
4. Pipeline orchestration layer
- autowisp/processing_steps/ is the operational layer that sequences the lower-level algorithms into named pipeline steps such as calibrate, find_stars, solve_astrometry, fit_magnitudes, create_lightcurves, epd, and tfa.
- autowisp/processor.py and autowisp/run_pipeline.py suggest the command/runtime entry points that execute those steps as a pipeline.
- This is the boundary between “science code” and “workflow control”.
5. Persistent storage and schema layer
- autowisp/database/ defines the project’s persistent model, centered on HDF5-backed structures rather than a traditional relational database.
- The data_model/ package contains the schema for images, targets, processing sequences, provenance, light-curve status/progress, master files, and HDF5 structure/version metadata.
- This layer is important architecturally because it is the source of truth for pipeline state, file relationships, and reproducibility.
6. File formats and I/O boundaries
- FITS is the primary raw/astronomical image format, handled by utilities such as fits_utilities.py, hdf5_file.py, file_utilities.py, and the hat/ parsers.
- HDF5 is the internal structured storage format for pipeline products and metadata.
- This creates a clear split between external telescope/camera data and internal normalized pipeline state.
7. Browser interface / operator console
- autowisp/browser_interface/ is a multi-app Django project used to manage and inspect pipeline projects.
- It is divided into functional apps:
- home for project creation/selection and filesystem navigation
- configuration for survey/project setup
- processing for running and monitoring pipeline steps
- results for light-curve/model review
- diagnostics for plotting and quality-control views
- core for shared UI templates, filesystem walking, and shared plotting helpers
- Middleware such as set_project_middleware.py and error_capture_middleware.py indicates the UI is project-scoped and error-aware.
8. Diagnostics and visualization path
- autowisp/diagnostics/ and autowisp/browser_interface/diagnostics/ provide image- and light-curve-level diagnostic generation.
- The browser UI includes templates and JS/CSS for progress views, review screens, plot displays, and calibrated-image previews.
- This is a read/inspect boundary: it consumes pipeline outputs but does not define the scientific pipeline itself.
9. Astrometry and external catalog integration
- autowisp/astrometry/ contains coordinate transforms and plate-solving support, including an astrometry_net_client, which implies interaction with external astrometry services.
- The README also emphasizes GAIA-based astrometry and sky-coordinate projection onto detector coordinates.
- Architecturally, this module is the bridge between image pixel space and catalog/star-identification space.
10. Photometry, detrending, and correction stack
- autowisp/magnitude_fitting/ performs ensemble fitting against reference stars and includes iterative refitting logic.
- autowisp/light_curves/ builds light curves and applies corrections, including EPD and TFA variants.
- These are the “science output refinement” stages after source measurements have already been made.
11. Command-line, scripts, and automation
- The scripts/ directory provides operational tooling for configuration parsing, database automation, plotting, exporting light curves, and generating parser code.
- autowisp/Dump2hdf5.py, error_cli.py, and the various helper scripts indicate the project is meant to be run both interactively and from automation/CLI workflows.
- This is an important deployment boundary: operators can run the pipeline headlessly or through the browser UI.
12. Testing, fixtures, and documentation
- The repo has extensive tests under autowisp/tests/ plus top-level tests, with sample FITS/HDF5 data and notebook-based demos.
- Documentation is substantial and includes generated API docs, design docs, and pipeline diagrams, which strongly suggests architecture is intentionally documented and versioned.
- Build/tooling uses pyproject.toml, Meson build files in many subpackages, GitHub Actions for tests, and Docker for packaging/runtime environments.
13. Key architectural boundaries to show in a diagram
- Input boundary: raw observatory FITS data and configuration files
- Processing boundary: stateless/scientific algorithms vs stateful workflow orchestration
- Persistence boundary: HDF5-backed database/state and generated products
- Presentation boundary: Django browser UI and plots
- External-service boundary: catalog/astrometry services and any referenced survey data
- Output boundary: calibrated images, master frames, per-star photometry, light curves, and detrended products
Main entry points: processing_steps Step orchestration · django_project Browser app · image_calibration Calibration · astrometry Astrometry
Components 21
Core pipeline
Scientific processing, orchestration, and data-flow stages.
- Calibration · Overscan correction, master frame creation, and calibrated image generation.
autowisp/image_calibration - Source find · Detects stars and prepares source-extracted measurements.
autowisp/source_finder.py - Astrometry · Transforms detector coordinates to sky/catalog coordinates and matches sources.
autowisp/astrometry - Photometry · Fits star shapes and measures aperture/PSF-PRF fluxes.
autowisp/processing_steps/measure_aperture_photometry.py - Mag fit · Builds ensemble magnitudes and refits photometric reference models.
autowisp/magnitude_fitting - Light curves · Collects per-star time series and applies light-curve corrections.
autowisp/light_curves - Detrending · Systematics removal and science-product refinement for light curves.
autowisp/processing_steps/lc_detrending.py - Step orchestration · Named pipeline steps that sequence the scientific algorithms and persist outputs.
autowisp/processing_steps
Persistence
HDF5-backed state, schemas, and project metadata.
- Data model · Schema and provenance structures for images, targets, runs, steps, and products.
autowisp/database/data_model - Database API · Database initialization, persistence helpers, and light-curve processing state.
autowisp/database
Browser UI
Django operator console and review tools.
- Browser app · Project-scoped Django console for configuration, processing, results, and diagnostics.
autowisp/browser_interface/django_project - Home · Project creation, selection, and filesystem navigation.
autowisp/browser_interface/home - Config · Survey and project configuration management.
autowisp/browser_interface/configuration - Processing UI · Launches steps, shows progress, logs, and review screens.
autowisp/browser_interface/processing - Results · Light-curve and model inspection tools.
autowisp/browser_interface/results - Diagnostics UI · Diagnostic plots and calibrated-image previews for quality control.
autowisp/browser_interface/diagnostics
I/O and external
File-format boundaries and external catalog/service integration.
- Raw FITS · Observatory FITS images and survey inputs feeding the pipeline.
- FITS/HDF5 I/O · Utilities for FITS parsing, HDF5 files, and internal product serialization.
autowisp/fits_utilities.py - Catalogs · Catalog and astrometry service integration, including GAIA/astrometry.net-style lookup.
autowisp/astrometry/astrometry_net_client.py
Connections 29
- Raw FITS → Calibration: calibrate · Raw images are corrected and stacked into calibrated/master products.
- Calibration → Source find: detect stars · Calibrated frames feed star detection and source extraction.
- Source find → Astrometry: match sky · Detected sources are matched to catalog coordinates and solved.
- Astrometry → Photometry: measure flux · Astrometrically identified sources are used for photometric fitting.
- Photometry → Mag fit: fit magnitudes · Per-image measurements are combined into ensemble magnitude solutions.
- Mag fit → Light curves: assemble curves · Fitted magnitudes are collected into time-series light curves.
- Light curves → Detrending: correct systematics · Light curves are detrended with EPD and TFA methods.
- Step orchestration → Calibration: step · Workflow orchestration invokes calibration steps.
- Step orchestration → Source find: step · Workflow orchestration invokes star-finding steps.
- Step orchestration → Astrometry: step · Workflow orchestration invokes astrometry solving.
- Step orchestration → Photometry: step · Workflow orchestration invokes photometry steps.
- Step orchestration → Mag fit: step · Workflow orchestration invokes magnitude fitting.
- Step orchestration → Light curves: step · Workflow orchestration invokes light-curve creation.
- Step orchestration → Detrending: step · Workflow orchestration invokes detrending and correction steps.
- Runner → Step orchestration: runs · The runtime executes the named processing sequence.
- Runner → Database API: persists state · Pipeline execution reads and writes structured processing state.
- Database API → Data model: implements · Database helpers are backed by the HDF5 data model and provenance schema.
- FITS/HDF5 I/O → Database API: serializes · FITS and HDF5 file boundaries feed database/product persistence.
- Catalogs → Astrometry: external refs · Astrometry relies on external catalog and solving services.
- Browser app → Home: includes · The Django project contains the home app.
- Browser app → Config: includes · The Django project contains the configuration app.
- Browser app → Processing UI: includes · The Django project contains the processing app.
- Browser app → Results: includes · The Django project contains the results app.
- Browser app → Diagnostics UI: includes · The Django project contains the diagnostics app.
- Processing UI → Step orchestration: controls · The UI launches and monitors pipeline steps.
- Results → Light curves: reviews · Results views consume light-curve products.
- Diagnostics UI → Calibration: inspects · Diagnostics views preview calibrated image products.
- Scripts → Runner: automation · Scripts drive headless pipeline execution and helper workflows.
- Scripts → Database API: admin · Scripts also manipulate database contents and exports.