How coderanirban71/aditya-l1-solar-flare-nowcasting-and-forecasting-model works
1. This repo is organized around a single end-to-end solar flare pipeline: ingest Aditya-L1 FITS telemetry, detect flare candidates with physics-based rules, label them against external flare catalogs, train an LSTM forecasting model, then serve both detection and prediction through a live dashboard.
2. The offline data-processing path lives in src/ and is staged as four sequential batch steps:
- data_ingestion.py builds a unified master_data.csv from raw HEL1OS and SoLEXS satellite files.
- statistical_trigger.py filters the stream with a Poisson significance test to produce triggered_data.csv.
- morphological_filter.py applies flare-shape validation to generate flare_catalog.csv.
- benchmarking.py matches detected events against GOES/NASA DONKI-style references to produce labeled training data.
3. src/pipeline.py is the orchestration boundary for that offline pipeline. Its role is to run the four preprocessing stages in order and materialize the intermediate datasets into outputs/. This is the main data-prep boundary between raw instrument data and everything downstream.
4. The detection logic is split into two conceptual engines. nowcast.py is the real-time flare detector that operates on current telemetry and significance thresholds, while forecast.py loads the trained model and emits future flare probabilities. Together they form the online inference layer.
5. The physics-first detection subsystem is important architecturally because it gates the ML layer. The system does not rely on the neural network alone; it first uses statistical triggering and morphological constraints to reject noise, glitches, and non-flare spikes before labeling an event as a candidate flare.
6. The ML training path is separated from runtime inference. src/forecast.ipynb is the training notebook, and the trained artifacts are stored under outputs/ as metadata and model/scaler files. That means training is an offline, GPU-friendly workflow, while the serving side only needs inference-capable dependencies.
7. The serving layer is src/server.py, which exposes the application state and data through REST endpoints. It appears to be stateful, maintaining replay position, play/pause, speed, and seek controls so the dashboard can behave like a time-based simulation rather than a passive chart viewer.
8. The API is the bridge between backend computation and visualization. Endpoints provide:
- current flux/timestamp/progress,
- nowcast alert state and flare class,
- forecast probability and upcoming events,
- chart-ready historical slices,
- the detected flare catalog,
- and replay controls such as play, pause, reset, speed, and seek.
9. The dashboard/ folder is a static frontend built with HTML, CSS, and JavaScript. It renders live plots, probability indicators, alert cards, and event logs, likely polling the FastAPI backend rather than pushing via websockets. This keeps the UI decoupled from the inference logic.
10. outputs/model_meta.json is a key runtime configuration artifact. It likely defines the model’s feature schema, thresholds, and evaluation metadata so the server can interpret the forecast model consistently without hardcoding training details in the API layer.
11. The main external data boundary is the PRADAN portal for Level-1 Aditya-L1 data, with NASA DONKI used as a supplementary labeling source during benchmarking. Those sources are only needed in the preprocessing/labeling stages, not for dashboard operation once the derived CSVs and model artifacts exist.
12. Tooling and runtime are split by purpose: Python for all backend/data work, PyTorch for model training and inference, FastAPI/Uvicorn for serving, and standard browser technologies for the UI. The repo also distinguishes between a general Python environment and a separate GPU training environment, which reflects the separation between offline model development and online deployment.
Main entry points: model_meta.json Outputs · server.py API server · pipeline.py Offline pipeline · benchmarking.py Benchmark events
Components 16
Offline pipeline
Batch preprocessing, labeling, and model preparation for solar flare data.
- Ingest data · Builds unified master telemetry data from raw instrument files.
src/data_ingestion.py - Trigger filter · Applies statistical significance testing to isolate candidate events.
src/statistical_trigger.py - Shape filter · Validates flare morphology and rejects non-flare spikes.
src/morphological_filter.py - Benchmark events · Matches detected events against external references to create labeled data.
src/benchmarking.py - Offline pipeline · Runs preprocessing stages in order and materializes intermediate datasets.
src/pipeline.py - Outputs · Derived CSVs, model metadata, and trained assets used downstream.
outputs/model_meta.json - Train model · Offline LSTM training workflow for flare forecasting.
src/forecast.ipynb - Model meta · Serialized schema, thresholds, and evaluation metadata for inference.
outputs/model_meta.json
Online serving
Real-time detection, forecasting, API state, and dashboard presentation.
- Nowcast · Detects active flare candidates from current telemetry using significance rules.
src/nowcast.py - Forecast · Loads trained artifacts and emits future flare probabilities.
src/forecast.py - API server · Maintains replay state and exposes telemetry, alerts, catalog, and controls.
src/server.py - Dashboard · Browser UI for live plots, alerts, probabilities, and event logs.
dashboard/index.html - UI logic · Fetches backend state and renders charts and controls.
dashboard/app.js - UI styles · Presentation layer for the dashboard.
dashboard/style.css
External sources
Upstream instrument and reference data used during preprocessing and benchmarking.
- PRADAN · Aditya-L1 Level-1 satellite data source.
- NASA DONKI · Reference flare events used for benchmarking and labeling.
Connections 20
- PRADAN → Ingest data: raw telemetry · Feeds Level-1 Aditya-L1 files into preprocessing.
- Ingest data → Trigger filter: master data · Passes unified stream to statistical triggering.
- Trigger filter → Shape filter: candidates · Hands off statistically significant events for shape validation.
- Shape filter → Benchmark events: flare events · Validated candidates are matched to external references.
- NASA DONKI → Benchmark events: reference labels · Provides external flare catalog matches for benchmarking.
- Offline pipeline → Ingest data: orchestrates · Executes preprocessing stages in sequence.
- Offline pipeline → Trigger filter: orchestrates · Executes preprocessing stages in sequence.
- Offline pipeline → Shape filter: orchestrates · Executes preprocessing stages in sequence.
- Offline pipeline → Benchmark events: orchestrates · Executes preprocessing stages in sequence.
- Benchmark events → Outputs: labeled data · Materializes training-ready datasets and catalogs.
- Train model → Outputs: trained artifacts · Produces model files and metadata for serving.
- Outputs → Model meta: model config · Provides feature schema and thresholds to runtime.
- Outputs → Forecast: model files · Supplies trained forecast artifacts for inference.
- Outputs → API server: catalog/data · Supplies derived datasets and runtime assets.
- Nowcast → API server: alert state · Provides real-time detection status and flare class.
- Forecast → API server: probabilities · Provides future flare likelihoods to the API.
- API server → Dashboard: REST data · Serves state, charts, alerts, and controls to the UI.
- Dashboard → UI logic: bootstraps · HTML loads client-side application logic.
- Dashboard → UI styles: styles · HTML page uses the dashboard stylesheet.
- UI logic → API server: polls · Fetches live state and control responses from the backend.