# kpenev/autowisp architecture

> Architecture diagram of the GitHub repository kpenev/autowisp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kpenev/autowisp
- Repository: https://github.com/kpenev/autowisp

## Overview

1. **Overall shape**
   - AutoWISP is a Python-based astronomical photometry pipeline with two major faces:
     - a **core processing library** under `autowisp/`
     - a **Django browser interface** under `autowisp/browser_interface/`
   - The repository is organized around a workflow that turns raw FITS images into calibrated images, extracted measurements, fitted magnitudes, light curves, and finally detrended science products.

2. **Primary data flow**
   - The main pipeline path is: **raw FITS input → calibration/master-frame generation → astrometry/source matching → photometry/PSF-PRF fitting → ensemble magnitude fitting → light-curve assembly → EPD/TFA detrending**.
   - Each stage writes artifacts that downstream stages consume, so the system is not a single monolithic run; it is a sequence of persisted processing steps.

3. **Core image-processing subsystem**
   - `autowisp/image_calibration/`, `autowisp/astrometry/`, `autowisp/source_finder.py`, `autowisp/fit_expression/`, `autowisp/magnitude_fitting/`, and `autowisp/light_curves/` implement the scientific algorithms.
   - These modules handle the mathematically intensive parts: overscan correction, master bias/dark/flat creation, coordinate transformations, PSF/PRF fitting, aperture photometry, regression-based magnitude fitting, and light-curve corrections.

4. **Pipeline orchestration layer**
   - `autowisp/processing_steps/` is the operational layer that sequences the lower-level algorithms into named pipeline steps such as `calibrate`, `find_stars`, `solve_astrometry`, `fit_magnitudes`, `create_lightcurves`, `epd`, and `tfa`.
   - `autowisp/processor.py` and `autowisp/run_pipeline.py` suggest the command/runtime entry points that execute those steps as a pipeline.
   - This is the boundary between “science code” and “workflow control”.

5. **Persistent storage and schema layer**
   - `autowisp/database/` defines the project’s persistent model, centered on **HDF5-backed structures** rather than a traditional relational database.
   - The `data_model/` package contains the schema for images, targets, processing sequences, provenance, light-curve status/progress, master files, and HDF5 structure/version metadata.
   - This layer is important architecturally because it is the source of truth for pipeline state, file relationships, and reproducibility.

6. **File formats and I/O boundaries**
   - FITS is the primary raw/astronomical image format, handled by utilities such as `fits_utilities.py`, `hdf5_file.py`, `file_utilities.py`, and the `hat/` parsers.
   - HDF5 is the internal structured storage format for pipeline products and metadata.
   - This creates a clear split between **external telescope/camera data** and **internal normalized pipeline state**.

7. **Browser interface / operator console**
   - `autowisp/browser_interface/` is a multi-app Django project used to manage and inspect pipeline projects.
   - It is divided into functional apps:
     - `home` for project creation/selection and filesystem navigation
     - `configuration` for survey/project setup
     - `processing` for running and monitoring pipeline steps
     - `results` for light-curve/model review
     - `diagnostics` for plotting and quality-control views
     - `core` for shared UI templates, filesystem walking, and shared plotting helpers
   - Middleware such as `set_project_middleware.py` and `error_capture_middleware.py` indicates the UI is project-scoped and error-aware.

8. **Diagnostics and visualization path**
   - `autowisp/diagnostics/` and `autowisp/browser_interface/diagnostics/` provide image- and light-curve-level diagnostic generation.
   - The browser UI includes templates and JS/CSS for progress views, review screens, plot displays, and calibrated-image previews.
   - This is a read/inspect boundary: it consumes pipeline outputs but does not define the scientific pipeline itself.

9. **Astrometry and external catalog integration**
   - `autowisp/astrometry/` contains coordinate transforms and plate-solving support, including an `astrometry_net_client`, which implies interaction with external astrometry services.
   - The README also emphasizes GAIA-based astrometry and sky-coordinate projection onto detector coordinates.
   - Architecturally, this module is the bridge between image pixel space and catalog/star-identification space.

10. **Photometry, detrending, and correction stack**
   - `autowisp/magnitude_fitting/` performs ensemble fitting against reference stars and includes iterative refitting logic.
   - `autowisp/light_curves/` builds light curves and applies corrections, including EPD and TFA variants.
   - These are the “science output refinement” stages after source measurements have already been made.

11. **Command-line, scripts, and automation**
   - The `scripts/` directory provides operational tooling for configuration parsing, database automation, plotting, exporting light curves, and generating parser code.
   - `autowisp/Dump2hdf5.py`, `error_cli.py`, and the various helper scripts indicate the project is meant to be run both interactively and from automation/CLI workflows.
   - This is an important deployment boundary: operators can run the pipeline headlessly or through the browser UI.

12. **Testing, fixtures, and documentation**
   - The repo has extensive tests under `autowisp/tests/` plus top-level tests, with sample FITS/HDF5 data and notebook-based demos.
   - Documentation is substantial and includes generated API docs, design docs, and pipeline diagrams, which strongly suggests architecture is intentionally documented and versioned.
   - Build/tooling uses `pyproject.toml`, Meson build files in many subpackages, GitHub Actions for tests, and Docker for packaging/runtime environments.

13. **Key architectural boundaries to show in a diagram**
   - **Input boundary:** raw observatory FITS data and configuration files
   - **Processing boundary:** stateless/scientific algorithms vs stateful workflow orchestration
   - **Persistence boundary:** HDF5-backed database/state and generated products
   - **Presentation boundary:** Django browser UI and plots
   - **External-service boundary:** catalog/astrometry services and any referenced survey data
   - **Output boundary:** calibrated images, master frames, per-star photometry, light curves, and detrended products

Main entry points: [`autowisp/processing_steps`](https://github.com/kpenev/autowisp/tree/master/autowisp/processing_steps) (Step orchestration), [`autowisp/browser_interface/django_project`](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/django_project) (Browser app), [`autowisp/image_calibration`](https://github.com/kpenev/autowisp/tree/master/autowisp/image_calibration) (Calibration), [`autowisp/astrometry`](https://github.com/kpenev/autowisp/tree/master/autowisp/astrometry) (Astrometry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core pipeline"]
  node_calibration["Calibration<br/>image prep"]
  node_source_find["Source find<br/>star extraction<br/>[source_finder.py]"]
  node_astrometry["Astrometry<br/>plate solving"]
  node_photometry["Photometry<br/>PSF/aperture fitting"]
  node_magfit["Mag fit<br/>ensemble fitting"]
  node_lightcurves["Light curves<br/>curve assembly"]
  node_detrending["Detrending<br/>EPD/TFA<br/>[lc_detrending.py]"]
  node_proc_steps["Step orchestration<br/>workflow layer"]
end

subgraph group_storage["Persistence"]
  node_db_model[("Data model<br/>HDF5 schema")]
  node_db_api[("Database API<br/>HDF5 store")]
end

subgraph group_ui["Browser UI"]
  node_browser["Browser app<br/>Django project"]
  node_home_app["Home<br/>project setup"]
  node_config_app["Config<br/>survey setup"]
  node_processing_app["Processing UI<br/>run monitor"]
  node_results_app["Results<br/>curve review"]
  node_diagnostics_app["Diagnostics UI<br/>qc views"]
end

subgraph group_io["I/O and external"]
  node_raw_input["Raw FITS<br/>astronomical input"]
  node_fits_io["FITS/HDF5 I/O<br/>file boundary<br/>[fits_utilities.py]"]
  node_catalog["Catalogs<br/>external service"]
end

subgraph group_ops["Automation"]
  node_processor(("Runner<br/>pipeline runtime<br/>[processor.py]"))
  node_scripts["Scripts<br/>automation tools"]
end

node_raw_input -->|"calibrate"| node_calibration
node_calibration -->|"detect stars"| node_source_find
node_source_find -->|"match sky"| node_astrometry
node_astrometry -->|"measure flux"| node_photometry
node_photometry -->|"fit magnitudes"| node_magfit
node_magfit -->|"assemble curves"| node_lightcurves
node_lightcurves -->|"correct systematics"| node_detrending
node_proc_steps -->|"step"| node_calibration
node_proc_steps -->|"step"| node_source_find
node_proc_steps -->|"step"| node_astrometry
node_proc_steps -->|"step"| node_photometry
node_proc_steps -->|"step"| node_magfit
node_proc_steps -->|"step"| node_lightcurves
node_proc_steps -->|"step"| node_detrending
node_processor -->|"runs"| node_proc_steps
node_processor -->|"persists state"| node_db_api
node_db_api -->|"implements"| node_db_model
node_fits_io -->|"serializes"| node_db_api
node_catalog -.->|"external refs"| node_astrometry
node_browser -->|"includes"| node_home_app
node_browser -->|"includes"| node_config_app
node_browser -->|"includes"| node_processing_app
node_browser -->|"includes"| node_results_app
node_browser -->|"includes"| node_diagnostics_app
node_processing_app -->|"controls"| node_proc_steps
node_results_app -->|"reviews"| node_lightcurves
node_diagnostics_app -->|"inspects"| node_calibration
node_scripts -->|"automation"| node_processor
node_scripts -->|"admin"| node_db_api

click node_calibration "https://github.com/kpenev/autowisp/tree/master/autowisp/image_calibration"
click node_source_find "https://github.com/kpenev/autowisp/blob/master/autowisp/source_finder.py"
click node_astrometry "https://github.com/kpenev/autowisp/tree/master/autowisp/astrometry"
click node_photometry "https://github.com/kpenev/autowisp/blob/master/autowisp/processing_steps/measure_aperture_photometry.py"
click node_magfit "https://github.com/kpenev/autowisp/tree/master/autowisp/magnitude_fitting"
click node_lightcurves "https://github.com/kpenev/autowisp/tree/master/autowisp/light_curves"
click node_detrending "https://github.com/kpenev/autowisp/blob/master/autowisp/processing_steps/lc_detrending.py"
click node_proc_steps "https://github.com/kpenev/autowisp/tree/master/autowisp/processing_steps"
click node_processor "https://github.com/kpenev/autowisp/blob/master/autowisp/processor.py"
click node_db_model "https://github.com/kpenev/autowisp/tree/master/autowisp/database/data_model"
click node_db_api "https://github.com/kpenev/autowisp/tree/master/autowisp/database"
click node_fits_io "https://github.com/kpenev/autowisp/blob/master/autowisp/fits_utilities.py"
click node_catalog "https://github.com/kpenev/autowisp/blob/master/autowisp/astrometry/astrometry_net_client.py"
click node_browser "https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/django_project"
click node_home_app "https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/home"
click node_config_app "https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/configuration"
click node_processing_app "https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/processing"
click node_results_app "https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/results"
click node_diagnostics_app "https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/diagnostics"
click node_scripts "https://github.com/kpenev/autowisp/tree/master/scripts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_calibration,node_source_find,node_astrometry,node_photometry,node_magfit,node_lightcurves,node_detrending,node_proc_steps toneBlue
class node_db_model,node_db_api toneAmber
class node_browser,node_home_app,node_config_app,node_processing_app,node_results_app,node_diagnostics_app toneMint
class node_raw_input,node_fits_io,node_catalog toneRose
class node_processor,node_scripts toneIndigo
```

## Components

### Core pipeline

Scientific processing, orchestration, and data-flow stages.

- [Calibration](https://github.com/kpenev/autowisp/tree/master/autowisp/image_calibration): image prep · Overscan correction, master frame creation, and calibrated image generation. · `autowisp/image_calibration`
- [Source find](https://github.com/kpenev/autowisp/blob/master/autowisp/source_finder.py): star extraction · Detects stars and prepares source-extracted measurements. · `autowisp/source_finder.py`
- [Astrometry](https://github.com/kpenev/autowisp/tree/master/autowisp/astrometry): plate solving · Transforms detector coordinates to sky/catalog coordinates and matches sources. · `autowisp/astrometry`
- [Photometry](https://github.com/kpenev/autowisp/blob/master/autowisp/processing_steps/measure_aperture_photometry.py): PSF/aperture fitting · Fits star shapes and measures aperture/PSF-PRF fluxes. · `autowisp/processing_steps/measure_aperture_photometry.py`
- [Mag fit](https://github.com/kpenev/autowisp/tree/master/autowisp/magnitude_fitting): ensemble fitting · Builds ensemble magnitudes and refits photometric reference models. · `autowisp/magnitude_fitting`
- [Light curves](https://github.com/kpenev/autowisp/tree/master/autowisp/light_curves): curve assembly · Collects per-star time series and applies light-curve corrections. · `autowisp/light_curves`
- [Detrending](https://github.com/kpenev/autowisp/blob/master/autowisp/processing_steps/lc_detrending.py): EPD/TFA · Systematics removal and science-product refinement for light curves. · `autowisp/processing_steps/lc_detrending.py`
- [Step orchestration](https://github.com/kpenev/autowisp/tree/master/autowisp/processing_steps): workflow layer · Named pipeline steps that sequence the scientific algorithms and persist outputs. · `autowisp/processing_steps`

### Persistence

HDF5-backed state, schemas, and project metadata.

- [Data model](https://github.com/kpenev/autowisp/tree/master/autowisp/database/data_model): HDF5 schema · Schema and provenance structures for images, targets, runs, steps, and products. · `autowisp/database/data_model`
- [Database API](https://github.com/kpenev/autowisp/tree/master/autowisp/database): HDF5 store · Database initialization, persistence helpers, and light-curve processing state. · `autowisp/database`

### Browser UI

Django operator console and review tools.

- [Browser app](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/django_project): Django project · Project-scoped Django console for configuration, processing, results, and diagnostics. · `autowisp/browser_interface/django_project`
- [Home](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/home): project setup · Project creation, selection, and filesystem navigation. · `autowisp/browser_interface/home`
- [Config](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/configuration): survey setup · Survey and project configuration management. · `autowisp/browser_interface/configuration`
- [Processing UI](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/processing): run monitor · Launches steps, shows progress, logs, and review screens. · `autowisp/browser_interface/processing`
- [Results](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/results): curve review · Light-curve and model inspection tools. · `autowisp/browser_interface/results`
- [Diagnostics UI](https://github.com/kpenev/autowisp/tree/master/autowisp/browser_interface/diagnostics): qc views · Diagnostic plots and calibrated-image previews for quality control. · `autowisp/browser_interface/diagnostics`

### I/O and external

File-format boundaries and external catalog/service integration.

- **Raw FITS**: astronomical input · Observatory FITS images and survey inputs feeding the pipeline.
- [FITS/HDF5 I/O](https://github.com/kpenev/autowisp/blob/master/autowisp/fits_utilities.py): file boundary · Utilities for FITS parsing, HDF5 files, and internal product serialization. · `autowisp/fits_utilities.py`
- [Catalogs](https://github.com/kpenev/autowisp/blob/master/autowisp/astrometry/astrometry_net_client.py): external service · Catalog and astrometry service integration, including GAIA/astrometry.net-style lookup. · `autowisp/astrometry/astrometry_net_client.py`

### Automation

CLI entry points, scripts, and headless workflows.

- [Runner](https://github.com/kpenev/autowisp/blob/master/autowisp/processor.py): pipeline runtime · Command/runtime entry points that execute the pipeline end to end. · `autowisp/processor.py`
- [Scripts](https://github.com/kpenev/autowisp/tree/master/scripts): automation tools · Operational scripts for config parsing, exports, plotting, and database automation. · `scripts`

## Connections

- Raw FITS → Calibration: calibrate · Raw images are corrected and stacked into calibrated/master products.
- Calibration → Source find: detect stars · Calibrated frames feed star detection and source extraction.
- Source find → Astrometry: match sky · Detected sources are matched to catalog coordinates and solved.
- Astrometry → Photometry: measure flux · Astrometrically identified sources are used for photometric fitting.
- Photometry → Mag fit: fit magnitudes · Per-image measurements are combined into ensemble magnitude solutions.
- Mag fit → Light curves: assemble curves · Fitted magnitudes are collected into time-series light curves.
- Light curves → Detrending: correct systematics · Light curves are detrended with EPD and TFA methods.
- Step orchestration → Calibration: step · Workflow orchestration invokes calibration steps.
- Step orchestration → Source find: step · Workflow orchestration invokes star-finding steps.
- Step orchestration → Astrometry: step · Workflow orchestration invokes astrometry solving.
- Step orchestration → Photometry: step · Workflow orchestration invokes photometry steps.
- Step orchestration → Mag fit: step · Workflow orchestration invokes magnitude fitting.
- Step orchestration → Light curves: step · Workflow orchestration invokes light-curve creation.
- Step orchestration → Detrending: step · Workflow orchestration invokes detrending and correction steps.
- Runner → Step orchestration: runs · The runtime executes the named processing sequence.
- Runner → Database API: persists state · Pipeline execution reads and writes structured processing state.
- Database API → Data model: implements · Database helpers are backed by the HDF5 data model and provenance schema.
- FITS/HDF5 I/O → Database API: serializes · FITS and HDF5 file boundaries feed database/product persistence.
- Catalogs → Astrometry: external refs · Astrometry relies on external catalog and solving services.
- Browser app → Home: includes · The Django project contains the home app.
- Browser app → Config: includes · The Django project contains the configuration app.
- Browser app → Processing UI: includes · The Django project contains the processing app.
- Browser app → Results: includes · The Django project contains the results app.
- Browser app → Diagnostics UI: includes · The Django project contains the diagnostics app.
- Processing UI → Step orchestration: controls · The UI launches and monitors pipeline steps.
- Results → Light curves: reviews · Results views consume light-curve products.
- Diagnostics UI → Calibration: inspects · Diagnostics views preview calibrated image products.
- Scripts → Runner: automation · Scripts drive headless pipeline execution and helper workflows.
- Scripts → Database API: admin · Scripts also manipulate database contents and exports.
