# sp-uhh/sgmse architecture

> Architecture diagram of the GitHub repository sp-uhh/sgmse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sp-uhh/sgmse
- Repository: https://github.com/sp-uhh/sgmse

## Overview

1. **Repository purpose and core loop**
   - This is a PyTorch speech restoration codebase for **speech enhancement and dereverberation** using **diffusion / score-based generative models** in the **complex STFT domain**.
   - The central flow is: **wav pairs in → spectrogram-based training/inference → iterative reverse diffusion → enhanced wav out**.

2. **Top-level entry points**
   - `train.py` is the training orchestrator.
   - `enhancement.py` runs checkpoint-based inference and writes enhanced audio files.
   - `calc_metrics.py` computes objective evaluation metrics from reference/enhanced wavs.
   - These scripts define the main user-facing boundaries: training, generation, and scoring are separate phases.

3. **Data boundary and dataset layout**
   - The repo expects a filesystem dataset organized as `train/`, `valid/`, and optionally `test/`, each with matching `clean/` and `noisy/` subfolders.
   - Data is WAV-only according to the README.
   - `sgmse/data_module.py` is the main bridge between disk data and the training loop, likely handling batching, splits, and transforms.

4. **Signal representation layer**
   - The model works on **time-frequency spectrograms**, not raw waveforms.
   - The README indicates tunable STFT-related parameters such as `n_fft`, `hop_length`, `spec_factor`, and `spec_abs_exponent`, so the front-end includes a nontrivial spectrogram scaling/normalization stage.
   - This representation layer is a key architectural boundary because all diffusion modeling and backbones operate on these spectral tensors.

5. **Diffusion / SDE modeling core**
   - `sgmse/sdes.py` defines the stochastic process side of the model: the forward noising process and its reverse-time formulation.
   - This is the mathematical heart of the repo: training learns to estimate scores or data-related targets under a chosen SDE, while sampling reconstructs clean speech by integrating the reverse dynamics.
   - The README references multiple SDE modes, including a Schrödinger-bridge-style setup, which suggests the SDE choice is configurable and materially changes the model behavior.

6. **Neural backbone subsystem**
   - `sgmse/backbones/` contains the denoising networks used to parameterize the score/denoiser:
     - `dcunet.py` for the Interspeech 2022 configuration
     - `ncsnpp.py`, `ncsnpp_v2.py`, and `ncsnpp_48k.py` for newer variants
   - `ncsnpp_utils/` contains shared architectural components such as up/downsampling, normalization, and custom CUDA/C++ ops.
   - The backbones are the main learnable function approximators, while the SDE defines what they are approximating.

7. **Sampling / reverse-process subsystem**
   - `sgmse/sampling/` contains `predictors.py` and `correctors.py`, which split the reverse generation procedure into prediction and correction steps.
   - This matches score-based sampling practice: a predictor advances the state along the reverse dynamics, and a corrector refines it with additional stochastic steps.
   - For architecture diagrams, this is the runtime generation path used during enhancement, distinct from the training objective.

8. **Model composition boundary**
   - `sgmse/model.py` is the composition layer that ties together:
     - the chosen backbone,
     - the selected SDE,
     - the loss formulation,
     - and the sampling/inference logic.
   - This is the main abstraction you would diagram as “the diffusion model,” with the other packages supplying its pluggable parts.
   - The README’s examples show that changing `--backbone`, `--sde`, and `--loss_type` alters this composition without changing the outer scripts.

9. **Inference utilities and audio I/O**
   - `sgmse/util/inference.py` and `enhancement.py` together define the inference boundary from checkpoint to enhanced WAVs.
   - `sgmse/util/tensors.py` and `sgmse/util/other.py` likely hold tensor helpers, reshaping, numerical utilities, and audio/spectrogram bookkeeping.
   - `sgmse/util/registry.py` suggests a configuration/registration mechanism for selecting backbones or SDE variants by name.

10. **Evaluation pipeline**
   - The evaluation path is explicitly two-stage:
     1. generate enhanced audio into `--enhanced_dir`
     2. score it with `calc_metrics.py`
   - This keeps synthesis and metric computation decoupled, which is useful for large batch evaluation and reproducibility.
   - The metrics script likely compares clean reference audio against enhanced outputs and reports standard speech quality/intelligibility scores.

11. **Preprocessing scripts and dataset construction**
   - `preprocessing/create_wsj0_chime3.py`, `create_wsj0_qut.py`, and `create_wsj0_reverb.py` prepare benchmark datasets from source corpora.
   - These scripts form a separate ingestion pipeline from training, producing the file layout expected by `train.py`.
   - The repo therefore has an offline preprocessing stage that sits before the model training loop.

12. **Logging, checkpoints, and experiment management**
   - `logs/` is the local checkpoint/log output area referenced by the README.
   - The repository supports Weights & Biases by default, with local CSV logging as a fallback.
   - This means experiment state is externalized through logger outputs and checkpoints, and `enhancement.py --ckpt` consumes those saved models directly.

13. **Technology and implementation footprint**
   - The project is a **Python 3.11 / PyTorch** codebase with custom low-level extensions for the NCSN++ backbone (`upfirdn2d.cpp`, `upfirdn2d_kernel.cu`).
   - The presence of `pyproject.toml`, `requirements.txt`, and `requirements_version.txt` indicates standard Python packaging plus pinned-environment support.
   - The CUDA/C++ ops are an important architectural detail because they affect the runtime path and backbone performance.

14. **What an architecture diagram should emphasize**
   - Show three main flows: **data preparation**, **training**, and **inference/evaluation**.
   - Place `sgmse/model.py` at the center, fed by `data_module.py`, `sdes.py`, and `backbones/`, and driving `sampling/` during generation.
   - Keep preprocessing, logging/checkpointing, and metric computation as surrounding support systems rather than part of the core model.

Main entry points: [`sgmse/backbones`](https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones) (Backbones), [`sgmse/model.py`](https://github.com/sp-uhh/sgmse/blob/main/sgmse/model.py) (Model), [`sgmse/util/inference.py`](https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/inference.py) (Inference utils), [`sgmse/data_module.py`](https://github.com/sp-uhh/sgmse/blob/main/sgmse/data_module.py) (Data module)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Entry points"]
  node_n1["Train<br/>orchestrator<br/>[train.py]"]
  node_n2["Enhance<br/>inference CLI<br/>[enhancement.py]"]
  node_n3["Metrics<br/>evaluation CLI<br/>[calc_metrics.py]"]
end

subgraph group_g2["Data pipeline"]
  node_n4["Prep data<br/>dataset builders"]
  node_n5["Data module<br/>batching bridge<br/>[data_module.py]"]
  node_n6{{"Spectrograms<br/>STFT front-end"}}
end

subgraph group_g3["Diffusion core"]
  node_n7["Model<br/>diffusion composition<br/>[model.py]"]
  node_n8["SDEs<br/>stochastic process<br/>[sdes.py]"]
  node_n9["Sampling<br/>reverse process"]
  node_n10["Predictors<br/>sampler step<br/>[predictors.py]"]
  node_n11["Correctors<br/>sampler step<br/>[correctors.py]"]
end

subgraph group_g4["Backbones"]
  node_n12["Backbones<br/>denoisers"]
  node_n13["DCUNet<br/>backbone<br/>[dcunet.py]"]
  node_n14["NCSN++<br/>backbone family<br/>[ncsnpp.py]"]
  node_n15["NCSN++ v2<br/>backbone variant<br/>[ncsnpp_v2.py]"]
  node_n16["48k NCSN++<br/>backbone variant<br/>[ncsnpp_48k.py]"]
  node_n17["Shared ops<br/>arch utilities"]
  node_n18{{"CUDA ops<br/>native extensions"}}
end

subgraph group_g5["Support"]
  node_n19["Inference utils<br/>runtime helpers<br/>[inference.py]"]
  node_n20["Registry<br/>selection helper<br/>[registry.py]"]
  node_n21["Tensor utils<br/>helpers<br/>[tensors.py]"]
  node_n22["Other utils<br/>helpers<br/>[other.py]"]
  node_n23[("Logs<br/>artifacts")]
end

node_n4 -->|"creates data"| node_n5
node_n5 -->|"emits tensors"| node_n6
node_n1 -->|"loads batches"| node_n5
node_n5 -->|"feeds model"| node_n7
node_n7 -->|"uses SDE"| node_n8
node_n7 -->|"selects backbone"| node_n12
node_n12 -->|"variant"| node_n13
node_n12 -->|"variant"| node_n14
node_n12 -->|"variant"| node_n15
node_n12 -->|"variant"| node_n16
node_n14 -->|"uses"| node_n17
node_n15 -->|"uses"| node_n17
node_n16 -->|"uses"| node_n17
node_n17 -->|"accelerates"| node_n18
node_n1 -->|"writes"| node_n23
node_n2 -->|"loads checkpoint"| node_n19
node_n19 -->|"runs model"| node_n7
node_n2 -->|"samples"| node_n9
node_n9 -->|"predict step"| node_n10
node_n9 -->|"correct step"| node_n11
node_n3 -->|"reads outputs"| node_n23
node_n7 -.->|"configured by"| node_n20
node_n19 -.->|"uses"| node_n21
node_n19 -.->|"uses"| node_n22

click node_n1 "https://github.com/sp-uhh/sgmse/blob/main/train.py"
click node_n2 "https://github.com/sp-uhh/sgmse/blob/main/enhancement.py"
click node_n3 "https://github.com/sp-uhh/sgmse/blob/main/calc_metrics.py"
click node_n4 "https://github.com/sp-uhh/sgmse/tree/main/preprocessing"
click node_n5 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/data_module.py"
click node_n7 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/model.py"
click node_n8 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/sdes.py"
click node_n9 "https://github.com/sp-uhh/sgmse/tree/main/sgmse/sampling"
click node_n10 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/sampling/predictors.py"
click node_n11 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/sampling/correctors.py"
click node_n12 "https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones"
click node_n13 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/dcunet.py"
click node_n14 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/ncsnpp.py"
click node_n15 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/ncsnpp_v2.py"
click node_n16 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/ncsnpp_48k.py"
click node_n17 "https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones/ncsnpp_utils"
click node_n18 "https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones/ncsnpp_utils/op"
click node_n19 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/inference.py"
click node_n20 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/registry.py"
click node_n21 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/tensors.py"
click node_n22 "https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/other.py"
click node_n23 "https://github.com/sp-uhh/sgmse/tree/main/logs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3 toneBlue
class node_n4,node_n5,node_n6 toneAmber
class node_n7,node_n8,node_n9,node_n10,node_n11 toneMint
class node_n12,node_n13,node_n14,node_n15,node_n16,node_n17,node_n18 toneRose
class node_n19,node_n20,node_n21,node_n22,node_n23 toneIndigo
```

## Components

### Entry points

User-facing scripts for training, enhancement, and evaluation.

- [Train](https://github.com/sp-uhh/sgmse/blob/main/train.py): orchestrator · Main training entry point. · `train.py`
- [Enhance](https://github.com/sp-uhh/sgmse/blob/main/enhancement.py): inference CLI · Loads a checkpoint and writes enhanced audio. · `enhancement.py`
- [Metrics](https://github.com/sp-uhh/sgmse/blob/main/calc_metrics.py): evaluation CLI · Computes objective scores for enhanced audio. · `calc_metrics.py`

### Data pipeline

Dataset preparation and disk-to-tensor loading.

- [Prep data](https://github.com/sp-uhh/sgmse/tree/main/preprocessing): dataset builders · Offline scripts that create benchmark dataset layouts. · `preprocessing`
- [Data module](https://github.com/sp-uhh/sgmse/blob/main/sgmse/data_module.py): batching bridge · Loads WAV pairs, handles splits, and feeds training batches. · `sgmse/data_module.py`
- **Spectrograms**: STFT front-end · Complex STFT representation and scaling/normalization boundary.

### Diffusion core

Model composition, SDEs, and sampling dynamics.

- [Model](https://github.com/sp-uhh/sgmse/blob/main/sgmse/model.py): diffusion composition · Wires backbone, SDE, loss, and sampling into one model. · `sgmse/model.py`
- [SDEs](https://github.com/sp-uhh/sgmse/blob/main/sgmse/sdes.py): stochastic process · Forward noising and reverse-time dynamics definitions. · `sgmse/sdes.py`
- [Sampling](https://github.com/sp-uhh/sgmse/tree/main/sgmse/sampling): reverse process · Predictor-corrector generation path used for enhancement. · `sgmse/sampling`
- [Predictors](https://github.com/sp-uhh/sgmse/blob/main/sgmse/sampling/predictors.py): sampler step · Advances the reverse diffusion state. · `sgmse/sampling/predictors.py`
- [Correctors](https://github.com/sp-uhh/sgmse/blob/main/sgmse/sampling/correctors.py): sampler step · Refines samples with additional stochastic updates. · `sgmse/sampling/correctors.py`

### Backbones

Learned denoisers and their shared architectural utilities.

- [Backbones](https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones): denoisers · Pluggable neural networks that parameterize the score or denoiser. · `sgmse/backbones`
- [DCUNet](https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/dcunet.py): backbone · Older backbone variant for the Interspeech 2022 setup. · `sgmse/backbones/dcunet.py`
- [NCSN++](https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/ncsnpp.py): backbone family · Main family of score-based backbones and variants. · `sgmse/backbones/ncsnpp.py`
- [NCSN++ v2](https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/ncsnpp_v2.py): backbone variant · Updated NCSN++ implementation. · `sgmse/backbones/ncsnpp_v2.py`
- [48k NCSN++](https://github.com/sp-uhh/sgmse/blob/main/sgmse/backbones/ncsnpp_48k.py): backbone variant · High-sample-rate backbone variant. · `sgmse/backbones/ncsnpp_48k.py`
- [Shared ops](https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones/ncsnpp_utils): arch utilities · Common layers, normalization, and up/downsampling helpers. · `sgmse/backbones/ncsnpp_utils`
- [CUDA ops](https://github.com/sp-uhh/sgmse/tree/main/sgmse/backbones/ncsnpp_utils/op): native extensions · Custom upfirdn2d C++/CUDA kernels used by backbone utilities. · `sgmse/backbones/ncsnpp_utils/op`

### Support

Inference helpers, logging, and experiment artifacts.

- [Inference utils](https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/inference.py): runtime helpers · Checkpoint loading and enhancement-time orchestration helpers. · `sgmse/util/inference.py`
- [Registry](https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/registry.py): selection helper · Name-to-component registration for configurable parts. · `sgmse/util/registry.py`
- [Tensor utils](https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/tensors.py): helpers · Tensor reshaping and numerical utility functions. · `sgmse/util/tensors.py`
- [Other utils](https://github.com/sp-uhh/sgmse/blob/main/sgmse/util/other.py): helpers · Miscellaneous shared utilities and bookkeeping. · `sgmse/util/other.py`
- [Logs](https://github.com/sp-uhh/sgmse/tree/main/logs): artifacts · Checkpoint and experiment output location. · `logs`

## Connections

- Prep data → Data module: creates data · Preprocessing scripts produce the dataset layout consumed by training.
- Data module → Spectrograms: emits tensors · Disk WAV pairs are transformed into spectrogram-domain features.
- Train → Data module: loads batches · Training reads data through the data module.
- Data module → Model: feeds model · Batches and spectral features enter the diffusion model.
- Model → SDEs: uses SDE · Model behavior depends on the selected stochastic process.
- Model → Backbones: selects backbone · The model instantiates a configurable denoiser network.
- Backbones → DCUNet: variant · DCUNet is one supported backbone choice.
- Backbones → NCSN++: variant · NCSN++ is a supported backbone family.
- Backbones → NCSN++ v2: variant · NCSN++ v2 is a supported backbone choice.
- Backbones → 48k NCSN++: variant · 48k NCSN++ is a supported backbone choice.
- NCSN++ → Shared ops: uses · NCSN++ relies on shared architectural utilities.
- NCSN++ v2 → Shared ops: uses · NCSN++ v2 relies on shared architectural utilities.
- 48k NCSN++ → Shared ops: uses · 48k NCSN++ relies on shared architectural utilities.
- Shared ops → CUDA ops: accelerates · Shared utilities call custom native kernels for performance.
- Train → Logs: writes · Training produces checkpoints and logs.
- Enhance → Inference utils: loads checkpoint · Enhancement uses inference helpers to restore a trained model.
- Inference utils → Model: runs model · Inference reconstructs speech through the composed diffusion model.
- Enhance → Sampling: samples · Enhancement invokes the reverse-process sampler.
- Sampling → Predictors: predict step · Sampling includes predictor updates.
- Sampling → Correctors: correct step · Sampling includes corrector updates.
- Metrics → Logs: reads outputs · Metrics are computed from generated artifacts and references.
- Model → Registry: configured by · Registry supports pluggable component selection by name.
- Inference utils → Tensor utils: uses · Inference helpers rely on tensor utilities.
- Inference utils → Other utils: uses · Inference helpers rely on general utilities.
