# ml4gw/ml4gw architecture

> Architecture diagram of the GitHub repository ml4gw/ml4gw, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ml4gw/ml4gw
- Repository: https://github.com/ml4gw/ml4gw

## Overview

1. **What this repo is**
   - `ml4gw` is a PyTorch-based scientific library for gravitational-wave ML workflows, not an application server or standalone pipeline.
   - Its architecture is organized around reusable building blocks for signal generation, preprocessing, dataset loading, and neural-network modules.

2. **Core data flow**
   - Typical flow is: raw or simulated time-series data → signal-conditioning transforms → optional augmentation/rescaling → dataset wrappers → neural network models.
   - A second major flow is waveform synthesis: astrophysical parameters → waveform generators → injected or standalone strain-like tensors used in training/evaluation.

3. **Time-series transforms subsystem**
   - `ml4gw/transforms` is the main preprocessing layer.
   - It contains operations for decimation, heterodyning, IIR filtering, integration, Pearson correlation, spectral estimation, spectrograms, Q-transforms, spline interpolation, scaling, SNR rescaling, and whitening.
   - These modules are the main boundary between raw detector-like signals and model-ready tensors.

4. **Waveform generation subsystem**
   - `ml4gw/waveforms` generates synthetic gravitational-wave signals.
   - It splits into `cbc` (compact binary coalescence models such as TaylorF2, PhenomD, PhenomP) and `adhoc` (e.g. sine-Gaussian and ringdown-style signals).
   - `generator.py` and `conversion.py` suggest a parameterization layer that converts physical parameters into waveform tensors.

5. **Data loading subsystem**
   - `ml4gw/dataloading` provides training data adapters for different storage/streaming patterns.
   - `in_memory_dataset.py`, `hdf5_dataset.py`, and `chunked_dataset.py` imply support for small local datasets, large on-disk datasets, and chunked access for scalable training.
   - This is the main I/O boundary between persisted data and the model-training stack.

6. **Neural network components**
   - `ml4gw/nn` contains reusable model components rather than a single end-to-end model.
   - Subpackages include `resnet` for 1D/2D residual architectures, `autoencoder` for convolutional autoencoders, `ssm/s4d.py` for state-space sequence modeling, `streaming` for online inference helpers, and `norm.py` for normalization layers.
   - These modules are designed to consume preprocessed GW-like tensors produced by the transforms and datasets.

7. **Utilities and physics helpers**
   - `ml4gw/utils` contains low-level helpers such as interferometer-related utilities and slicing.
   - Top-level modules like `gw.py`, `spectral.py`, `augmentations.py`, `distributions.py`, `constants.py`, and `types.py` provide shared physics/math helpers, tensor utilities, and type definitions used across the package.

8. **Streaming / online inference boundary**
   - The `nn/streaming` package introduces explicit online-processing support through components like `snapshotter` and `online_average`.
   - This indicates the library is not limited to offline training; it also supports sliding-window or incremental inference over time series.

9. **Testing strategy**
   - `tests/` mirrors the package layout almost one-to-one, which makes the architecture easy to map and suggests strong module-level isolation.
   - There are dedicated test groups for dataloading, transforms, waveform synthesis, utilities, and each NN subsystem.
   - `benchmarks/` mirrors major functional areas as well, indicating the project tracks performance for compute-heavy signal-processing and model components.

10. **Documentation and examples**
   - `docs/` is a first-class part of the repo, with tutorials, installation docs, and example pages for augmentation, distributions, GW utilities, and transform families.
   - The notebooks under `docs/tutorials/` act as higher-level integration examples that likely demonstrate the intended end-to-end flow from signals to models.

11. **Packaging and release**
   - The project is packaged as a Python library via `pyproject.toml` and pinned with `uv.lock`.
   - GitHub Actions handles documentation builds, pre-commit checks, unit tests, and publishing, so the repo’s operational boundary is standard library CI/CD rather than runtime infrastructure.

12. **Architecture summary for a diagram**
   - The cleanest diagram would show three main capability layers: **signal generation**, **signal preprocessing/data access**, and **ML models/streaming inference**.
   - Shared utilities and spectral/physics helpers sit beneath those layers, while tests/docs/benchmarks wrap the whole codebase as validation and usability scaffolding.

Main entry points: [`ml4gw/transforms`](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/transforms) (Transforms), [`ml4gw/waveforms`](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms) (Waveforms), [`ml4gw/nn`](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn) (Neural nets), [`ml4gw`](https://github.com/ml4gw/ml4gw/tree/main/ml4gw) (Shared helpers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_generation["Signal generation"]
  node_n_waveforms["Waveforms<br/>signal synthesis"]
  node_n_cbc["CBC models<br/>waveform family"]
  node_n_adhoc["Ad hoc signals<br/>waveform family"]
  node_n_conversion["Conversion<br/>parameter map<br/>[conversion.py]"]
  node_n_generator["Generator<br/>waveform builder<br/>[generator.py]"]
end

subgraph group_g_preprocessing["Preprocessing and data"]
  node_n_transforms["Transforms<br/>preprocessing"]
  node_n_spectral["Spectral tools<br/>spectral ops<br/>[spectral.py]"]
  node_n_whitening["Whitening<br/>normalization<br/>[whitening.py]"]
  node_n_qtransform["Q transform<br/>time-frequency<br/>[qtransform.py]"]
  node_n_scaler["Scaling<br/>[scaler.py]"]
  node_n_dataloading["Datasets<br/>data access"]
  node_n_in_memory["In-memory<br/>dataset"]
  node_n_hdf5[("HDF5<br/>dataset<br/>[hdf5_dataset.py]")]
  node_n_chunked["Chunked<br/>dataset<br/>[chunked_dataset.py]"]
end

subgraph group_g_models["Models and inference"]
  node_n_nn["Neural nets<br/>model components"]
  node_n_resnet["ResNet<br/>feature extractor"]
  node_n_autoencoder["Autoencoder<br/>representation model"]
  node_n_s4d["S4D<br/>sequence model<br/>[s4d.py]"]
  node_n_streaming["Streaming<br/>online inference"]
end

subgraph group_g_shared["Shared core"]
  node_n_shared["Shared helpers<br/>core utilities"]
  node_n_gw["GW utils<br/>physics helpers<br/>[gw.py]"]
  node_n_utils["Utilities<br/>low-level utils"]
  node_n_augmentations["Augmentations<br/>data transforms<br/>[augmentations.py]"]
  node_n_distributions["Distributions<br/>random models<br/>[distributions.py]"]
end

subgraph group_g_validation["Validation and docs"]
  node_n_tests["Tests<br/>validation"]
  node_n_benchmarks["Benchmarks<br/>performance"]
  node_n_docs["Docs<br/>documentation"]
end

node_n_waveforms -->|"contains"| node_n_cbc
node_n_waveforms -->|"contains"| node_n_adhoc
node_n_waveforms -->|"uses"| node_n_conversion
node_n_waveforms -->|"exposes"| node_n_generator
node_n_generator -->|"parameterizes"| node_n_conversion
node_n_transforms -->|"includes"| node_n_spectral
node_n_transforms -->|"includes"| node_n_whitening
node_n_transforms -->|"includes"| node_n_qtransform
node_n_transforms -->|"includes"| node_n_scaler
node_n_dataloading -->|"contains"| node_n_in_memory
node_n_dataloading -->|"contains"| node_n_hdf5
node_n_dataloading -->|"contains"| node_n_chunked
node_n_nn -->|"contains"| node_n_resnet
node_n_nn -->|"contains"| node_n_autoencoder
node_n_nn -->|"contains"| node_n_s4d
node_n_nn -->|"contains"| node_n_streaming
node_n_streaming -->|"supports"| node_n_s4d
node_n_shared -->|"provides"| node_n_gw
node_n_shared -->|"provides"| node_n_utils
node_n_shared -->|"provides"| node_n_augmentations
node_n_shared -->|"provides"| node_n_distributions
node_n_transforms -->|"depends on"| node_n_shared
node_n_waveforms -->|"depends on"| node_n_shared
node_n_dataloading -->|"feeds"| node_n_transforms
node_n_transforms -->|"feeds"| node_n_nn
node_n_waveforms -->|"feeds"| node_n_transforms
node_n_tests -.->|"covers"| node_n_transforms
node_n_tests -.->|"covers"| node_n_waveforms
node_n_tests -.->|"covers"| node_n_dataloading
node_n_tests -.->|"covers"| node_n_nn
node_n_benchmarks -.->|"measures"| node_n_transforms
node_n_benchmarks -.->|"measures"| node_n_waveforms
node_n_benchmarks -.->|"measures"| node_n_nn
node_n_docs -.->|"documents"| node_n_transforms
node_n_docs -.->|"documents"| node_n_waveforms
node_n_docs -.->|"documents"| node_n_shared

click node_n_waveforms "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms"
click node_n_cbc "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms/cbc"
click node_n_adhoc "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms/adhoc"
click node_n_conversion "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/waveforms/conversion.py"
click node_n_generator "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/waveforms/generator.py"
click node_n_transforms "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/transforms"
click node_n_spectral "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/spectral.py"
click node_n_whitening "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/whitening.py"
click node_n_qtransform "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/qtransform.py"
click node_n_scaler "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/scaler.py"
click node_n_dataloading "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/dataloading"
click node_n_in_memory "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/dataloading/in_memory_dataset.py"
click node_n_hdf5 "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/dataloading/hdf5_dataset.py"
click node_n_chunked "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/dataloading/chunked_dataset.py"
click node_n_nn "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn"
click node_n_resnet "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn/resnet"
click node_n_autoencoder "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn/autoencoder"
click node_n_s4d "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/nn/ssm/s4d.py"
click node_n_streaming "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn/streaming"
click node_n_shared "https://github.com/ml4gw/ml4gw/tree/main/ml4gw"
click node_n_gw "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/gw.py"
click node_n_utils "https://github.com/ml4gw/ml4gw/tree/main/ml4gw/utils"
click node_n_augmentations "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/augmentations.py"
click node_n_distributions "https://github.com/ml4gw/ml4gw/blob/main/ml4gw/distributions.py"
click node_n_tests "https://github.com/ml4gw/ml4gw/tree/main/tests"
click node_n_benchmarks "https://github.com/ml4gw/ml4gw/tree/main/benchmarks"
click node_n_docs "https://github.com/ml4gw/ml4gw/tree/main/docs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_waveforms,node_n_cbc,node_n_adhoc,node_n_conversion,node_n_generator toneBlue
class node_n_transforms,node_n_spectral,node_n_whitening,node_n_qtransform,node_n_scaler,node_n_dataloading,node_n_in_memory,node_n_hdf5,node_n_chunked toneAmber
class node_n_nn,node_n_resnet,node_n_autoencoder,node_n_s4d,node_n_streaming toneMint
class node_n_shared,node_n_gw,node_n_utils,node_n_augmentations,node_n_distributions toneRose
class node_n_tests,node_n_benchmarks,node_n_docs toneIndigo
```

## Components

### Signal generation

Synthetic waveform and parameter-to-strain synthesis

- [Waveforms](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms): signal synthesis · Top-level waveform generation package · `ml4gw/waveforms`
- [CBC models](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms/cbc): waveform family · Compact binary coalescence waveform models · `ml4gw/waveforms/cbc`
- [Ad hoc signals](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/waveforms/adhoc): waveform family · Sine-Gaussian and ringdown-style synthetic signals · `ml4gw/waveforms/adhoc`
- [Conversion](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/waveforms/conversion.py): parameter map · Physical-parameter to waveform conversion layer · `ml4gw/waveforms/conversion.py`
- [Generator](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/waveforms/generator.py): waveform builder · Waveform assembly and generation entry point · `ml4gw/waveforms/generator.py`

### Preprocessing and data

Transforms and dataset adapters that prepare training tensors

- [Transforms](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/transforms): preprocessing · Main signal-conditioning and feature-extraction layer · `ml4gw/transforms`
- [Spectral tools](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/spectral.py): spectral ops · Spectral density and frequency-domain helpers · `ml4gw/transforms/spectral.py`
- [Whitening](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/whitening.py): normalization · Noise whitening and conditioning transforms · `ml4gw/transforms/whitening.py`
- [Q transform](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/qtransform.py): time-frequency · Q-transform feature extraction · `ml4gw/transforms/qtransform.py`
- [Scaling](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/transforms/scaler.py): rescaling · Amplitude and SNR rescaling utilities · `ml4gw/transforms/scaler.py`
- [Datasets](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/dataloading): data access · Training data adapters and storage backends · `ml4gw/dataloading`
- [In-memory](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/dataloading/in_memory_dataset.py): dataset · Small local dataset wrapper · `ml4gw/dataloading/in_memory_dataset.py`
- [HDF5](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/dataloading/hdf5_dataset.py): dataset · On-disk HDF5 dataset wrapper · `ml4gw/dataloading/hdf5_dataset.py`
- [Chunked](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/dataloading/chunked_dataset.py): dataset · Chunked access for scalable training · `ml4gw/dataloading/chunked_dataset.py`

### Models and inference

Reusable neural components and streaming inference helpers

- [Neural nets](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn): model components · Reusable neural network building blocks · `ml4gw/nn`
- [ResNet](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn/resnet): feature extractor · 1D and 2D residual architectures · `ml4gw/nn/resnet`
- [Autoencoder](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn/autoencoder): representation model · Convolutional autoencoder components · `ml4gw/nn/autoencoder`
- [S4D](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/nn/ssm/s4d.py): sequence model · State-space sequence modeling module · `ml4gw/nn/ssm/s4d.py`
- [Streaming](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/nn/streaming): online inference · Sliding-window and incremental inference helpers · `ml4gw/nn/streaming`

### Shared core

Package-wide physics, math, and utility helpers

- [Shared helpers](https://github.com/ml4gw/ml4gw/tree/main/ml4gw): core utilities · Package-wide physics, math, and type helpers · `ml4gw`
- [GW utils](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/gw.py): physics helpers · Gravitational-wave domain utilities · `ml4gw/gw.py`
- [Utilities](https://github.com/ml4gw/ml4gw/tree/main/ml4gw/utils): low-level utils · Interferometer and slicing helpers · `ml4gw/utils`
- [Augmentations](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/augmentations.py): data transforms · Shared augmentation utilities · `ml4gw/augmentations.py`
- [Distributions](https://github.com/ml4gw/ml4gw/blob/main/ml4gw/distributions.py): random models · Sampling and parameter distribution helpers · `ml4gw/distributions.py`

### Validation and docs

Tests, benchmarks, and documentation that mirror the library

- [Tests](https://github.com/ml4gw/ml4gw/tree/main/tests): validation · Module-level test suite mirroring the package layout · `tests`
- [Benchmarks](https://github.com/ml4gw/ml4gw/tree/main/benchmarks): performance · Performance coverage for compute-heavy components · `benchmarks`
- [Docs](https://github.com/ml4gw/ml4gw/tree/main/docs): documentation · User guides, examples, and tutorials · `docs`

## Connections

- Waveforms → CBC models: contains · Waveform family for compact binaries
- Waveforms → Ad hoc signals: contains · Waveform family for synthetic ad hoc signals
- Waveforms → Conversion: uses · Converts physical parameters into waveform tensors
- Waveforms → Generator: exposes · Builds waveform generation workflows
- Generator → Conversion: parameterizes · Maps source parameters into generator inputs
- Transforms → Spectral tools: includes · Frequency-domain estimation helpers
- Transforms → Whitening: includes · Noise conditioning layer
- Transforms → Q transform: includes · Time-frequency feature extraction
- Transforms → Scaling: includes · Amplitude and SNR rescaling
- Datasets → In-memory: contains · Small-dataset adapter
- Datasets → HDF5: contains · Persistent on-disk dataset adapter
- Datasets → Chunked: contains · Chunked streaming dataset adapter
- Neural nets → ResNet: contains · Residual feature extractors
- Neural nets → Autoencoder: contains · Convolutional representation models
- Neural nets → S4D: contains · State-space sequence model
- Neural nets → Streaming: contains · Online inference helpers
- Streaming → S4D: supports · Streaming-friendly sequence processing
- Shared helpers → GW utils: provides · Domain-specific gravitational-wave helpers
- Shared helpers → Utilities: provides · Low-level utility boundary
- Shared helpers → Augmentations: provides · Reusable augmentation helpers
- Shared helpers → Distributions: provides · Sampling and prior helpers
- Transforms → Shared helpers: depends on · Transforms use shared physics and utility helpers
- Waveforms → Shared helpers: depends on · Waveform code reuses common helpers
- Datasets → Transforms: feeds · Datasets supply tensors to preprocessing and training
- Transforms → Neural nets: feeds · Conditioned tensors flow into model components
- Waveforms → Transforms: feeds · Synthetic signals can be injected into preprocessing
- Tests → Transforms: covers · Mirrors transform modules
- Tests → Waveforms: covers · Mirrors waveform modules
- Tests → Datasets: covers · Mirrors dataset adapters
- Tests → Neural nets: covers · Mirrors neural network components
- Benchmarks → Transforms: measures · Benchmarks signal-processing routines
- Benchmarks → Waveforms: measures · Benchmarks waveform generation
- Benchmarks → Neural nets: measures · Benchmarks model components
- Docs → Transforms: documents · Examples and tutorials for preprocessing
- Docs → Waveforms: documents · Examples for waveform generation
- Docs → Shared helpers: documents · Examples for shared helpers and utilities