# geomagpy/magpy architecture

> Architecture diagram of the GitHub repository geomagpy/magpy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geomagpy/magpy
- Repository: https://github.com/geomagpy/magpy

## Overview

1. **Overall shape**
   - This repository is a Python package, not a standalone service.
   - The architectural center is a reusable time-series processing library for geomagnetic data, with optional CLI tools and a GUI on top.
   - Most functionality flows through one internal data model: a `DataStream` object.

2. **Core data model boundary**
   - `magpy.stream` defines the main stream abstraction.
   - A stream combines:
     - a numeric array for time-series samples
     - a metadata/header dictionary for sensor, format, and analysis context
   - Almost all subsystems read from, transform, or write back to this stream object rather than using separate domain models.

3. **I/O and format adapters**
   - `magpy/lib` contains the largest integration surface: one module per supported file or exchange format.
   - `stream.read()` and `DataStream.write()` route data through these format handlers.
   - Supported inputs/outputs include many geomagnetic and scientific formats such as IAGA-02, IMAGCDF, BLV, WDC, IAF, CSV, JSON, NetCDF-like variants, and several instrument-specific formats.
   - This layer is the main boundary between MagPy’s internal model and external data representations.

4. **Transformation and utility layer**
   - `magpy/core/conversions.py` and `magpy/opt/*` provide coordinate transforms, table helpers, numerical utilities, and supporting calculations.
   - This layer is used by analysis and format code to normalize coordinates, handle missing values, round times, parse headers, and convert between representations.
   - These are mostly shared helpers rather than top-level workflows.

5. **Time-series analysis subsystem**
   - `magpy/core/methods.py` and methods on `DataStream` implement the common signal-processing pipeline:
     - trimming, resampling, filtering, smoothing
     - interpolation and gap filling
     - statistics, offsets, scaling, derivatives, integration
     - joining/merging/subtracting streams
   - This is the main “processing engine” for data before downstream analysis or export.

6. **Geomagnetic-specific analysis**
   - `magpy/core/activity.py` adds domain-specific analysis:
     - K-index calculation
     - storm detection
     - Sq/base variation estimation
   - These algorithms consume normalized one-minute or second-resolution streams and often depend on filtering/resampling performed by the core stream layer.

7. **Flagging and annotation subsystem**
   - `magpy/core/flagging.py` is a separate subsystem for labeling ranges and events in time-series data.
   - Flags are stored as structured records with sensor IDs, time spans, components, types, labels, and comments.
   - Flagging can be generated from outliers, ranges, binary states, or converted from other datasets, then applied back to streams to filter or annotate them.
   - Flags can also be serialized to JSON or embedded in certain CDF-based outputs.

8. **DI / absolutes / baseline workflow**
   - `magpy/absolutes.py` plus support in `core/methods.py` implements the geomagnetic absolute measurement workflow.
   - This subsystem handles DI-flux measurements, basevalue extraction, pier differences, baseline fitting, and baseline correction.
   - Data flow here is more specialized:
     - read DI files and continuous variometer/scalar data
     - align/interpolate continuous measurements to DI timestamps
     - compute absolutes and basevalues
     - fit/apply baselines
     - export baseline and correction products

9. **Database integration**
   - `magpy/core/database.py` provides the persistence boundary to MySQL/MariaDB.
   - It stores and retrieves:
     - streams and data lines
     - sensor metadata
     - pier metadata and coordinate data
     - flags
     - absolute/basevalue records
     - baseline definitions
   - The database is optional, but when used it acts as a structured backend for long-lived observatory metadata and processed results.

10. **UI, CLI, examples, and verification**
   - `magpy/gui` contains the optional XMagPy desktop GUI, which sits on top of the same core library and is meant for interactive observatory workflows.
   - `magpy/scripts` contains command-line entry points such as `mpconvert` and `addcred`.
   - `magpy/examples` and `magpy/test` provide the reference data, regression checks, and format coverage used to validate the library.
   - `Dockerfile` and packaging files show the project is meant to be deployed as an installable Python environment, with Docker as an optional runtime container.

11. **Primary data flow for the whole system**
   - The dominant flow is:
     - ingest from file, web service, FTP, or database
     - normalize into `DataStream`
     - analyze/flag/fit/transform
     - optionally persist processed results or export in another format
   - The GUI and CLI do not form separate pipelines; they orchestrate the same library boundaries.

12. **Key architectural separation**
   - The codebase cleanly separates:
     - generic stream manipulation
     - geomagnetic domain algorithms
     - format adapters
     - metadata/flag persistence
     - interactive UI wrappers
   - That separation is what you would want to show in an architecture diagram: a central stream model with several specialized satellites around it, not a service-oriented or layered web stack.

Main entry points: [`magpy/stream.py`](https://github.com/geomagpy/magpy/blob/master/magpy/stream.py) (DataStream), [`magpy/lib/__init__.py`](https://github.com/geomagpy/magpy/blob/master/magpy/lib/__init__.py) (Format adapters), [`magpy/absolutes.py`](https://github.com/geomagpy/magpy/blob/master/magpy/absolutes.py) (Absolutes), [`magpy/core/methods.py`](https://github.com/geomagpy/magpy/blob/master/magpy/core/methods.py) (Methods)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core processing"]
  node_stream[("DataStream<br/>stream model<br/>[stream.py]")]
  node_methods["Methods<br/>processing engine<br/>[methods.py]"]
  node_activity["Activity<br/>geomagnetic analysis<br/>[activity.py]"]
  node_flagging["Flagging<br/>event labels<br/>[flagging.py]"]
  node_database[("Database<br/>mysql backend<br/>[database.py]")]
  node_absolutes["Absolutes<br/>DI/baseline workflow<br/>[absolutes.py]"]
  node_conversions["Conversions<br/>coordinate helpers<br/>[conversions.py]"]
  node_opt["Opt helpers<br/>utility toolkit<br/>[__init__.py]"]
end

subgraph group_io["I/O adapters"]
  node_lib["Format adapters<br/>format layer<br/>[__init__.py]"]
  node_formats["Format modules<br/>adapters<br/>[magpy_formats.py]"]
  node_absformats["Abs formats<br/>absolute adapters"]
end

subgraph group_ui["User entry points"]
  node_gui["XMagPy<br/>desktop GUI<br/>[xmagpy.py]"]
  node_scripts["CLI tools<br/>entry points<br/>[mpconvert.py]"]
end

subgraph group_support["Supporting material"]
  node_examples["Examples<br/>reference data"]
  node_tests["Tests<br/>regression suite"]
  node_docs["Docs<br/>[README.md]"]
end

node_methods -->|"operates on"| node_stream
node_activity -->|"builds on"| node_methods
node_flagging -->|"annotates"| node_stream
node_absolutes -->|"uses"| node_methods
node_absolutes -->|"reads/writes"| node_stream
node_database -->|"persists"| node_stream
node_database -->|"stores"| node_flagging
node_database -->|"stores"| node_absolutes
node_conversions -->|"supports"| node_methods
node_opt -->|"supports"| node_methods
node_lib -->|"reads/writes"| node_stream
node_lib -->|"dispatches to"| node_formats
node_lib -->|"dispatches to"| node_absformats
node_formats -->|"adapts"| node_stream
node_absformats -->|"adapts"| node_absolutes
node_gui -->|"orchestrates"| node_stream
node_scripts -->|"invokes"| node_lib
node_scripts -->|"invokes"| node_absolutes
node_tests -->|"validates"| node_lib
node_tests -->|"validates"| node_absolutes
node_examples -->|"exercises"| node_lib
node_docs -->|"describes"| node_stream

click node_stream "https://github.com/geomagpy/magpy/blob/master/magpy/stream.py"
click node_methods "https://github.com/geomagpy/magpy/blob/master/magpy/core/methods.py"
click node_activity "https://github.com/geomagpy/magpy/blob/master/magpy/core/activity.py"
click node_flagging "https://github.com/geomagpy/magpy/blob/master/magpy/core/flagging.py"
click node_database "https://github.com/geomagpy/magpy/blob/master/magpy/core/database.py"
click node_absolutes "https://github.com/geomagpy/magpy/blob/master/magpy/absolutes.py"
click node_conversions "https://github.com/geomagpy/magpy/blob/master/magpy/core/conversions.py"
click node_opt "https://github.com/geomagpy/magpy/blob/master/magpy/opt/__init__.py"
click node_lib "https://github.com/geomagpy/magpy/blob/master/magpy/lib/__init__.py"
click node_formats "https://github.com/geomagpy/magpy/blob/master/magpy/lib/magpy_formats.py"
click node_absformats "https://github.com/geomagpy/magpy/blob/master/magpy/lib/magpy_absformats.py"
click node_gui "https://github.com/geomagpy/magpy/blob/master/magpy/gui/xmagpy.py"
click node_scripts "https://github.com/geomagpy/magpy/blob/master/magpy/scripts/mpconvert.py"
click node_examples "https://github.com/geomagpy/magpy/tree/master/magpy/examples"
click node_tests "https://github.com/geomagpy/magpy/tree/master/magpy/test"
click node_docs "https://github.com/geomagpy/magpy/blob/master/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_stream,node_methods,node_activity,node_flagging,node_database,node_absolutes,node_conversions,node_opt toneBlue
class node_lib,node_formats,node_absformats toneAmber
class node_gui,node_scripts toneMint
class node_examples,node_tests,node_docs toneRose
```

## Components

### Core processing

Stream model, shared algorithms, geomagnetic analysis, and persistence boundary.

- [DataStream](https://github.com/geomagpy/magpy/blob/master/magpy/stream.py): stream model · Central time-series container combining samples with metadata and headers. · `magpy/stream.py`
- [Methods](https://github.com/geomagpy/magpy/blob/master/magpy/core/methods.py): processing engine · Core stream operations for trimming, resampling, filtering, interpolation, joins, statistics, and transforms. · `magpy/core/methods.py`
- [Activity](https://github.com/geomagpy/magpy/blob/master/magpy/core/activity.py): geomagnetic analysis · Domain algorithms such as K-index, storm detection, and Sq/base variation estimation. · `magpy/core/activity.py`
- [Flagging](https://github.com/geomagpy/magpy/blob/master/magpy/core/flagging.py): event labels · Structured flag records and helpers for outliers, ranges, states, serialization, and application to streams. · `magpy/core/flagging.py`
- [Database](https://github.com/geomagpy/magpy/blob/master/magpy/core/database.py): mysql backend · Optional persistence layer for streams, metadata, flags, absolutes, and baseline records. · `magpy/core/database.py`
- [Absolutes](https://github.com/geomagpy/magpy/blob/master/magpy/absolutes.py): DI/baseline workflow · DI-flux, basevalue, baseline fitting, and correction workflow for observatory calibration. · `magpy/absolutes.py`
- [Conversions](https://github.com/geomagpy/magpy/blob/master/magpy/core/conversions.py): coordinate helpers · Normalization and representation helpers for coordinates, headers, times, and values. · `magpy/core/conversions.py`
- [Opt helpers](https://github.com/geomagpy/magpy/blob/master/magpy/opt/__init__.py): utility toolkit · Shared utility modules for tables, checks, significant figures, and supporting calculations. · `magpy/opt/__init__.py`

### I/O adapters

Format handlers and adapter registries for external data exchange.

- [Format adapters](https://github.com/geomagpy/magpy/blob/master/magpy/lib/__init__.py): format layer · Central registry and implementation surface for file and exchange formats. · `magpy/lib/__init__.py`
- [Format modules](https://github.com/geomagpy/magpy/blob/master/magpy/lib/magpy_formats.py): adapters · Representative per-format readers and writers such as IAGA-02, CDF, JSON, CSV, WDC, and instrument-specific variants. · `magpy/lib/magpy_formats.py`
- [Abs formats](https://github.com/geomagpy/magpy/blob/master/magpy/lib/magpy_absformats.py): absolute adapters · Specialized adapters for absolute-measurement and baseline-related file formats. · `magpy/lib/magpy_absformats.py`

### User entry points

Optional GUI and command-line wrappers around the library.

- [XMagPy](https://github.com/geomagpy/magpy/blob/master/magpy/gui/xmagpy.py): desktop GUI · Optional interactive desktop interface for observatory workflows and analysis. · `magpy/gui/xmagpy.py`
- [CLI tools](https://github.com/geomagpy/magpy/blob/master/magpy/scripts/mpconvert.py): entry points · Command-line wrappers such as conversion and credential management utilities. · `magpy/scripts/mpconvert.py`

### Supporting material

Examples, tests, and project packaging artifacts.

- [Examples](https://github.com/geomagpy/magpy/tree/master/magpy/examples): reference data · Packaged sample files used to demonstrate formats and workflows. · `magpy/examples`
- [Tests](https://github.com/geomagpy/magpy/tree/master/magpy/test): regression suite · Automated checks covering reading, DI workflow, baselines, and format handling. · `magpy/test`
- [Docs](https://github.com/geomagpy/magpy/blob/master/README.md): package docs · Manuals, notebooks, and repository documentation for usage and architecture. · `README.md`

## Connections

- Methods → DataStream: operates on · Most processing functions read and return DataStream objects.
- Activity → Methods: builds on · Domain analysis depends on core resampling, filtering, and normalization.
- Flagging → DataStream: annotates · Flags are applied back onto streams to label or filter intervals.
- Absolutes → Methods: uses · Absolute-measurement workflows reuse interpolation, alignment, and baseline helpers.
- Absolutes → DataStream: reads/writes · DI and baseline processing consumes and produces streams and derived products.
- Database → DataStream: persists · Streams and data lines are stored and retrieved through the database boundary.
- Database → Flagging: stores · Flag records can be persisted alongside observatory data.
- Database → Absolutes: stores · Absolute and baseline records are part of the persistence model.
- Conversions → Methods: supports · Shared conversion helpers feed the core processing pipeline.
- Opt helpers → Methods: supports · Utility routines assist numeric handling and table-style operations.
- Format adapters → DataStream: reads/writes · The main I/O boundary routes external formats into DataStream and back out.
- Format adapters → Format modules: dispatches to · Registry modules coordinate the per-format readers and writers.
- Format adapters → Abs formats: dispatches to · Absolute-measurement exchange formats are handled by dedicated adapters.
- Format modules → DataStream: adapts · Format handlers normalize external representations into the internal stream model.
- Abs formats → Absolutes: adapts · Specialized adapters support baseline and DI workflow files.
- XMagPy → DataStream: orchestrates · The desktop UI drives the same underlying library workflows.
- CLI tools → Format adapters: invokes · CLI entry points call the library for conversion and related tasks.
- CLI tools → Absolutes: invokes · Command-line tools can trigger absolute and calibration workflows.
- Tests → Format adapters: validates · Regression tests cover format read/write behavior and integration paths.
- Tests → Absolutes: validates · Dedicated tests cover DI and baseline workflows.
- Examples → Format adapters: exercises · Packaged sample files demonstrate and cover the supported adapters.
- Docs → DataStream: describes · Documentation explains the stream-centered architecture and workflows.