How geomagpy/magpy works
1. Overall shape
- This repository is a Python package, not a standalone service.
- The architectural center is a reusable time-series processing library for geomagnetic data, with optional CLI tools and a GUI on top.
- Most functionality flows through one internal data model: a DataStream object.
2. Core data model boundary
- magpy.stream defines the main stream abstraction.
- A stream combines:
- a numeric array for time-series samples
- a metadata/header dictionary for sensor, format, and analysis context
- Almost all subsystems read from, transform, or write back to this stream object rather than using separate domain models.
3. I/O and format adapters
- magpy/lib contains the largest integration surface: one module per supported file or exchange format.
- stream.read() and DataStream.write() route data through these format handlers.
- Supported inputs/outputs include many geomagnetic and scientific formats such as IAGA-02, IMAGCDF, BLV, WDC, IAF, CSV, JSON, NetCDF-like variants, and several instrument-specific formats.
- This layer is the main boundary between MagPy’s internal model and external data representations.
4. Transformation and utility layer
- magpy/core/conversions.py and magpy/opt/* provide coordinate transforms, table helpers, numerical utilities, and supporting calculations.
- This layer is used by analysis and format code to normalize coordinates, handle missing values, round times, parse headers, and convert between representations.
- These are mostly shared helpers rather than top-level workflows.
5. Time-series analysis subsystem
- magpy/core/methods.py and methods on DataStream implement the common signal-processing pipeline:
- trimming, resampling, filtering, smoothing
- interpolation and gap filling
- statistics, offsets, scaling, derivatives, integration
- joining/merging/subtracting streams
- This is the main “processing engine” for data before downstream analysis or export.
6. Geomagnetic-specific analysis
- magpy/core/activity.py adds domain-specific analysis:
- K-index calculation
- storm detection
- Sq/base variation estimation
- These algorithms consume normalized one-minute or second-resolution streams and often depend on filtering/resampling performed by the core stream layer.
7. Flagging and annotation subsystem
- magpy/core/flagging.py is a separate subsystem for labeling ranges and events in time-series data.
- Flags are stored as structured records with sensor IDs, time spans, components, types, labels, and comments.
- Flagging can be generated from outliers, ranges, binary states, or converted from other datasets, then applied back to streams to filter or annotate them.
- Flags can also be serialized to JSON or embedded in certain CDF-based outputs.
8. DI / absolutes / baseline workflow
- magpy/absolutes.py plus support in core/methods.py implements the geomagnetic absolute measurement workflow.
- This subsystem handles DI-flux measurements, basevalue extraction, pier differences, baseline fitting, and baseline correction.
- Data flow here is more specialized:
- read DI files and continuous variometer/scalar data
- align/interpolate continuous measurements to DI timestamps
- compute absolutes and basevalues
- fit/apply baselines
- export baseline and correction products
9. Database integration
- magpy/core/database.py provides the persistence boundary to MySQL/MariaDB.
- It stores and retrieves:
- streams and data lines
- sensor metadata
- pier metadata and coordinate data
- flags
- absolute/basevalue records
- baseline definitions
- The database is optional, but when used it acts as a structured backend for long-lived observatory metadata and processed results.
10. UI, CLI, examples, and verification
- magpy/gui contains the optional XMagPy desktop GUI, which sits on top of the same core library and is meant for interactive observatory workflows.
- magpy/scripts contains command-line entry points such as mpconvert and addcred.
- magpy/examples and magpy/test provide the reference data, regression checks, and format coverage used to validate the library.
- Dockerfile and packaging files show the project is meant to be deployed as an installable Python environment, with Docker as an optional runtime container.
11. Primary data flow for the whole system
- The dominant flow is:
- ingest from file, web service, FTP, or database
- normalize into DataStream
- analyze/flag/fit/transform
- optionally persist processed results or export in another format
- The GUI and CLI do not form separate pipelines; they orchestrate the same library boundaries.
12. Key architectural separation
- The codebase cleanly separates:
- generic stream manipulation
- geomagnetic domain algorithms
- format adapters
- metadata/flag persistence
- interactive UI wrappers
- That separation is what you would want to show in an architecture diagram: a central stream model with several specialized satellites around it, not a service-oriented or layered web stack.
Main entry points: stream.py DataStream · __init__.py Format adapters · absolutes.py Absolutes · methods.py Methods
Components 16
Core processing
Stream model, shared algorithms, geomagnetic analysis, and persistence boundary.
- DataStream · Central time-series container combining samples with metadata and headers.
magpy/stream.py - Methods · Core stream operations for trimming, resampling, filtering, interpolation, joins, statistics, and transforms.
magpy/core/methods.py - Activity · Domain algorithms such as K-index, storm detection, and Sq/base variation estimation.
magpy/core/activity.py - Flagging · Structured flag records and helpers for outliers, ranges, states, serialization, and application to streams.
magpy/core/flagging.py - Database · Optional persistence layer for streams, metadata, flags, absolutes, and baseline records.
magpy/core/database.py - Absolutes · DI-flux, basevalue, baseline fitting, and correction workflow for observatory calibration.
magpy/absolutes.py - Conversions · Normalization and representation helpers for coordinates, headers, times, and values.
magpy/core/conversions.py - Opt helpers · Shared utility modules for tables, checks, significant figures, and supporting calculations.
magpy/opt/__init__.py
I/O adapters
Format handlers and adapter registries for external data exchange.
- Format adapters · Central registry and implementation surface for file and exchange formats.
magpy/lib/__init__.py - Format modules · Representative per-format readers and writers such as IAGA-02, CDF, JSON, CSV, WDC, and instrument-specific variants.
magpy/lib/magpy_formats.py - Abs formats · Specialized adapters for absolute-measurement and baseline-related file formats.
magpy/lib/magpy_absformats.py
User entry points
Optional GUI and command-line wrappers around the library.
Supporting material
Examples, tests, and project packaging artifacts.
Connections 22
- Methods → DataStream: operates on · Most processing functions read and return DataStream objects.
- Activity → Methods: builds on · Domain analysis depends on core resampling, filtering, and normalization.
- Flagging → DataStream: annotates · Flags are applied back onto streams to label or filter intervals.
- Absolutes → Methods: uses · Absolute-measurement workflows reuse interpolation, alignment, and baseline helpers.
- Absolutes → DataStream: reads/writes · DI and baseline processing consumes and produces streams and derived products.
- Database → DataStream: persists · Streams and data lines are stored and retrieved through the database boundary.
- Database → Flagging: stores · Flag records can be persisted alongside observatory data.
- Database → Absolutes: stores · Absolute and baseline records are part of the persistence model.
- Conversions → Methods: supports · Shared conversion helpers feed the core processing pipeline.
- Opt helpers → Methods: supports · Utility routines assist numeric handling and table-style operations.
- Format adapters → DataStream: reads/writes · The main I/O boundary routes external formats into DataStream and back out.
- Format adapters → Format modules: dispatches to · Registry modules coordinate the per-format readers and writers.
- Format adapters → Abs formats: dispatches to · Absolute-measurement exchange formats are handled by dedicated adapters.
- Format modules → DataStream: adapts · Format handlers normalize external representations into the internal stream model.
- Abs formats → Absolutes: adapts · Specialized adapters support baseline and DI workflow files.
- XMagPy → DataStream: orchestrates · The desktop UI drives the same underlying library workflows.
- CLI tools → Format adapters: invokes · CLI entry points call the library for conversion and related tasks.
- CLI tools → Absolutes: invokes · Command-line tools can trigger absolute and calibration workflows.
- Tests → Format adapters: validates · Regression tests cover format read/write behavior and integration paths.
- Tests → Absolutes: validates · Dedicated tests cover DI and baseline workflows.
- Examples → Format adapters: exercises · Packaged sample files demonstrate and cover the supported adapters.
- Docs → DataStream: describes · Documentation explains the stream-centered architecture and workflows.