How ec-jrc/nrt works
1. What this repository is
- nrt is a Python library, not an application service.
- Its core purpose is near-real-time change detection in spatio-temporal satellite image time series.
- The architectural center is a stateful monitoring workflow: fit a model on a stable history period, then process one new acquisition at a time and flag breaks/anomalies.
2. Primary data model and boundary
- The package is built around array-based geospatial data, especially xarray DataArray/Dataset objects and NumPy arrays.
- Typical inputs are satellite cubes, masks, and time slices; typical outputs are break detections, monitoring state, reports, and serialized model state.
- This library boundary is important: it does not manage data ingestion pipelines itself, but expects already prepared time-series data and masks.
3. Main runtime flow
- The standard flow is:
1. select a stable history period
2. fit a model to the expected normal behavior
3. enter monitoring mode
4. ingest each new observation sequentially
5. compare observation vs predicted behavior
6. emit a break / disturbance signal when divergence persists
- The API is designed for online / incremental monitoring, which makes the monitor object stateful between acquisitions.
4. Core algorithm subsystem
- The nrt.monitor package contains the algorithm implementations:
- ewma.py — Exponentially Weighted Moving Average
- cusum.py — Cumulative Sum of residuals
- mosum.py — Moving Sum of residuals
- ccdc.py — Continuous Change Detection and Classification of land cover
- iqr.py — interquartile-range based outlier detection
- These algorithms share the same conceptual pipeline but differ in how residuals are accumulated, thresholds are computed, and breaks are confirmed.
- CCDC is explicitly described as a partial implementation of the published method.
5. Model fitting layer
- nrt/fit_methods.py provides the fitting routines used to build the “normal behavior” model.
- The README and example show a linear/harmonic trend model being fit on the history period before monitoring begins.
- This layer is the bridge between raw time series and monitoring algorithms: it defines the expected signal against which future observations are compared.
6. Shared analysis utilities
- stats.py, outliers.py, utils.py, and utils_efp.py contain shared numerical and helper logic used across monitors.
- These modules act as a common support layer for:
- statistical calculations
- residual handling
- thresholding / outlier decisions
- event or breakpoint-related utilities
- log.py provides logging support for operational use.
7. State, persistence, and operational continuity
- The monitors are designed to retain state across time steps, which matters in near-real-time operations.
- The README calls out two key operational outputs:
- report(...) for generating a monitoring report
- to_netcdf(...) for saving monitor state to disk
- NetCDF support is a meaningful architecture point because it allows stop/restart of long-running monitoring jobs and interoperates with geospatial data tooling.
8. Technology stack that shapes the design
- The package depends on:
- NumPy and SciPy for numeric/statistical computation
- xarray for labeled multi-dimensional data
- numba for performance on compute-heavy routines
- rasterio and netCDF4 for geospatial/raster and persistent scientific data formats
- This stack indicates the package is optimized for scientific array processing at scale, rather than general-purpose application logic.
9. Examples, docs, and distribution boundary
- The docs/ tree and gallery scripts demonstrate intended usage patterns, including parallel computing and simulation data.
- The package is documented with Read the Docs and distributed as a standard Python package via pyproject.toml.
- The architecture therefore includes a clear separation between library code, example workflows, and published API documentation.
10. Testing and validation structure
- tests/unit_tests validates the numerical helpers and fitting logic against fixture CSV data.
- tests/integration_tests exercises the monitor implementations end-to-end.
- This split reflects the architecture: lower-level math utilities are tested independently, while monitor classes are validated as operational pipelines.
11. What to draw in an architecture diagram
- A useful diagram would show:
- Satellite time-series inputs and masks feeding into
- a history fitting stage
- then into one of several monitoring algorithm modules
- with shared stats/utilities underneath
- and state/report persistence via NetCDF
- The most important boundary is that the system is a single-process analytical library with pluggable monitoring strategies, not a distributed service.
Main entry points: __init__.py Monitor API · stats.py Stats · fit_methods.py Fit models · unit_tests Unit tests
Components 18
Core library
Library boundary for fitting, monitoring, and shared numerical helpers.
- Satellite series · Prepared time-series cubes and masks used as monitor inputs.
- xarray/NumPy · Labeled arrays and ndarrays used throughout the library.
- Fit models · Builds the expected normal-behavior model from stable history.
nrt/fit_methods.py - Monitor API · Shared monitor interface and online processing flow.
nrt/monitor/__init__.py - EWMA · Exponentially weighted moving average monitor.
nrt/monitor/ewma.py - CUSUM · Cumulative-sum residual monitor.
nrt/monitor/cusum.py - MOSUM · Moving-sum residual monitor.
nrt/monitor/mosum.py - CCDC · Partial implementation of continuous change detection and classification.
nrt/monitor/ccdc.py - IQR · Interquartile-range based anomaly detection.
nrt/monitor/iqr.py - Stats · Shared statistical calculations used by fitting and monitors.
nrt/stats.py - Outliers · Residual and threshold logic for anomaly decisions.
nrt/outliers.py - Utils · Common helper routines shared across the package.
nrt/utils.py - EFP utils · Breakpoint and event-related utilities.
nrt/utils_efp.py - Logging · Operational logging support for library use.
nrt/log.py - State I/O · Report generation and NetCDF serialization for pause/resume monitoring.
Docs & examples
User-facing documentation and gallery workflows.
- Docs & gallery · Reference docs and example workflows, including parallel computing and simulation demos.
docs
Tests
Unit and integration validation of math helpers and monitors.
- Unit tests · Tests for fitting logic and shared numerical helpers.
tests/unit_tests - Integration tests · End-to-end monitor tests for operational behavior.
tests/integration_tests
Connections 25
- Satellite series → xarray/NumPy: stored as · Inputs are handled as labeled arrays and ndarrays.
- xarray/NumPy → Fit models: fit history · History-period arrays are used to fit the expected signal.
- Fit models → Monitor API: initialize · Fitted model state is passed into online monitoring.
- Monitor API → EWMA: strategy · EWMA is one monitoring implementation.
- Monitor API → CUSUM: strategy · CUSUM is one monitoring implementation.
- Monitor API → MOSUM: strategy · MOSUM is one monitoring implementation.
- Monitor API → CCDC: strategy · CCDC is one monitoring implementation.
- Monitor API → IQR: strategy · IQR is one monitoring implementation.
- EWMA → Stats: uses · Algorithms depend on shared statistics.
- CUSUM → Stats: uses · Algorithms depend on shared statistics.
- MOSUM → Stats: uses · Algorithms depend on shared statistics.
- CCDC → Stats: uses · Algorithms depend on shared statistics.
- IQR → Outliers: uses · Outlier decisions reuse shared threshold logic.
- Stats → Utils: supports · Shared helpers underpin numerical routines.
- Outliers → Utils: supports · Shared helpers underpin decision logic.
- EFP utils → Stats: supports · Breakpoint utilities feed statistical routines.
- Monitor API → State I/O: persist/report · Monitoring state can be reported and saved to NetCDF.
- State I/O → Logging: logs · Operational persistence and reporting are logged.
- Docs & gallery → Fit models: demonstrates · Documentation shows fitting workflows.
- Docs & gallery → Monitor API: demonstrates · Documentation shows monitoring workflows.
- Unit tests → Fit models: verifies · Unit tests cover fitting routines.
- Unit tests → Stats: verifies · Unit tests cover numerical helpers.
- Unit tests → Outliers: verifies · Unit tests cover outlier logic.
- Integration tests → Monitor API: verifies · Integration tests exercise end-to-end monitors.
- Integration tests → CCDC: verifies · CCDC has dedicated integration coverage.