How sp-uhh/sgmse works
1. Repository purpose and core loop
- This is a PyTorch speech restoration codebase for speech enhancement and dereverberation using diffusion / score-based generative models in the complex STFT domain.
- The central flow is: wav pairs in → spectrogram-based training/inference → iterative reverse diffusion → enhanced wav out.
2. Top-level entry points
- train.py is the training orchestrator.
- enhancement.py runs checkpoint-based inference and writes enhanced audio files.
- calc_metrics.py computes objective evaluation metrics from reference/enhanced wavs.
- These scripts define the main user-facing boundaries: training, generation, and scoring are separate phases.
3. Data boundary and dataset layout
- The repo expects a filesystem dataset organized as train/, valid/, and optionally test/, each with matching clean/ and noisy/ subfolders.
- Data is WAV-only according to the README.
- sgmse/data_module.py is the main bridge between disk data and the training loop, likely handling batching, splits, and transforms.
4. Signal representation layer
- The model works on time-frequency spectrograms, not raw waveforms.
- The README indicates tunable STFT-related parameters such as n_fft, hop_length, spec_factor, and spec_abs_exponent, so the front-end includes a nontrivial spectrogram scaling/normalization stage.
- This representation layer is a key architectural boundary because all diffusion modeling and backbones operate on these spectral tensors.
5. Diffusion / SDE modeling core
- sgmse/sdes.py defines the stochastic process side of the model: the forward noising process and its reverse-time formulation.
- This is the mathematical heart of the repo: training learns to estimate scores or data-related targets under a chosen SDE, while sampling reconstructs clean speech by integrating the reverse dynamics.
- The README references multiple SDE modes, including a Schrödinger-bridge-style setup, which suggests the SDE choice is configurable and materially changes the model behavior.
6. Neural backbone subsystem
- sgmse/backbones/ contains the denoising networks used to parameterize the score/denoiser:
- dcunet.py for the Interspeech 2022 configuration
- ncsnpp.py, ncsnpp_v2.py, and ncsnpp_48k.py for newer variants
- ncsnpp_utils/ contains shared architectural components such as up/downsampling, normalization, and custom CUDA/C++ ops.
- The backbones are the main learnable function approximators, while the SDE defines what they are approximating.
7. Sampling / reverse-process subsystem
- sgmse/sampling/ contains predictors.py and correctors.py, which split the reverse generation procedure into prediction and correction steps.
- This matches score-based sampling practice: a predictor advances the state along the reverse dynamics, and a corrector refines it with additional stochastic steps.
- For architecture diagrams, this is the runtime generation path used during enhancement, distinct from the training objective.
8. Model composition boundary
- sgmse/model.py is the composition layer that ties together:
- the chosen backbone,
- the selected SDE,
- the loss formulation,
- and the sampling/inference logic.
- This is the main abstraction you would diagram as “the diffusion model,” with the other packages supplying its pluggable parts.
- The README’s examples show that changing --backbone, --sde, and --loss_type alters this composition without changing the outer scripts.
9. Inference utilities and audio I/O
- sgmse/util/inference.py and enhancement.py together define the inference boundary from checkpoint to enhanced WAVs.
- sgmse/util/tensors.py and sgmse/util/other.py likely hold tensor helpers, reshaping, numerical utilities, and audio/spectrogram bookkeeping.
- sgmse/util/registry.py suggests a configuration/registration mechanism for selecting backbones or SDE variants by name.
10. Evaluation pipeline
- The evaluation path is explicitly two-stage:
1. generate enhanced audio into --enhanced_dir
2. score it with calc_metrics.py
- This keeps synthesis and metric computation decoupled, which is useful for large batch evaluation and reproducibility.
- The metrics script likely compares clean reference audio against enhanced outputs and reports standard speech quality/intelligibility scores.
11. Preprocessing scripts and dataset construction
- preprocessing/create_wsj0_chime3.py, create_wsj0_qut.py, and create_wsj0_reverb.py prepare benchmark datasets from source corpora.
- These scripts form a separate ingestion pipeline from training, producing the file layout expected by train.py.
- The repo therefore has an offline preprocessing stage that sits before the model training loop.
12. Logging, checkpoints, and experiment management
- logs/ is the local checkpoint/log output area referenced by the README.
- The repository supports Weights & Biases by default, with local CSV logging as a fallback.
- This means experiment state is externalized through logger outputs and checkpoints, and enhancement.py --ckpt consumes those saved models directly.
13. Technology and implementation footprint
- The project is a Python 3.11 / PyTorch codebase with custom low-level extensions for the NCSN++ backbone (upfirdn2d.cpp, upfirdn2d_kernel.cu).
- The presence of pyproject.toml, requirements.txt, and requirements_version.txt indicates standard Python packaging plus pinned-environment support.
- The CUDA/C++ ops are an important architectural detail because they affect the runtime path and backbone performance.
14. What an architecture diagram should emphasize
- Show three main flows: data preparation, training, and inference/evaluation.
- Place sgmse/model.py at the center, fed by data_module.py, sdes.py, and backbones/, and driving sampling/ during generation.
- Keep preprocessing, logging/checkpointing, and metric computation as surrounding support systems rather than part of the core model.
Main entry points: backbones Backbones · model.py Model · inference.py Inference utils · data_module.py Data module
Components 23
Entry points
User-facing scripts for training, enhancement, and evaluation.
Data pipeline
Dataset preparation and disk-to-tensor loading.
- Prep data · Offline scripts that create benchmark dataset layouts.
preprocessing - Data module · Loads WAV pairs, handles splits, and feeds training batches.
sgmse/data_module.py - Spectrograms · Complex STFT representation and scaling/normalization boundary.
Diffusion core
Model composition, SDEs, and sampling dynamics.
- Model · Wires backbone, SDE, loss, and sampling into one model.
sgmse/model.py - SDEs · Forward noising and reverse-time dynamics definitions.
sgmse/sdes.py - Sampling · Predictor-corrector generation path used for enhancement.
sgmse/sampling - Predictors · Advances the reverse diffusion state.
sgmse/sampling/predictors.py - Correctors · Refines samples with additional stochastic updates.
sgmse/sampling/correctors.py
Backbones
Learned denoisers and their shared architectural utilities.
- Backbones · Pluggable neural networks that parameterize the score or denoiser.
sgmse/backbones - DCUNet · Older backbone variant for the Interspeech 2022 setup.
sgmse/backbones/dcunet.py - NCSN++ · Main family of score-based backbones and variants.
sgmse/backbones/ncsnpp.py - NCSN++ v2 · Updated NCSN++ implementation.
sgmse/backbones/ncsnpp_v2.py - 48k NCSN++ · High-sample-rate backbone variant.
sgmse/backbones/ncsnpp_48k.py - Shared ops · Common layers, normalization, and up/downsampling helpers.
sgmse/backbones/ncsnpp_utils - CUDA ops · Custom upfirdn2d C++/CUDA kernels used by backbone utilities.
sgmse/backbones/ncsnpp_utils/op
Support
Inference helpers, logging, and experiment artifacts.
- Inference utils · Checkpoint loading and enhancement-time orchestration helpers.
sgmse/util/inference.py - Registry · Name-to-component registration for configurable parts.
sgmse/util/registry.py - Tensor utils · Tensor reshaping and numerical utility functions.
sgmse/util/tensors.py - Other utils · Miscellaneous shared utilities and bookkeeping.
sgmse/util/other.py - Logs · Checkpoint and experiment output location.
logs
Connections 24
- Prep data → Data module: creates data · Preprocessing scripts produce the dataset layout consumed by training.
- Data module → Spectrograms: emits tensors · Disk WAV pairs are transformed into spectrogram-domain features.
- Train → Data module: loads batches · Training reads data through the data module.
- Data module → Model: feeds model · Batches and spectral features enter the diffusion model.
- Model → SDEs: uses SDE · Model behavior depends on the selected stochastic process.
- Model → Backbones: selects backbone · The model instantiates a configurable denoiser network.
- Backbones → DCUNet: variant · DCUNet is one supported backbone choice.
- Backbones → NCSN++: variant · NCSN++ is a supported backbone family.
- Backbones → NCSN++ v2: variant · NCSN++ v2 is a supported backbone choice.
- Backbones → 48k NCSN++: variant · 48k NCSN++ is a supported backbone choice.
- NCSN++ → Shared ops: uses · NCSN++ relies on shared architectural utilities.
- NCSN++ v2 → Shared ops: uses · NCSN++ v2 relies on shared architectural utilities.
- 48k NCSN++ → Shared ops: uses · 48k NCSN++ relies on shared architectural utilities.
- Shared ops → CUDA ops: accelerates · Shared utilities call custom native kernels for performance.
- Train → Logs: writes · Training produces checkpoints and logs.
- Enhance → Inference utils: loads checkpoint · Enhancement uses inference helpers to restore a trained model.
- Inference utils → Model: runs model · Inference reconstructs speech through the composed diffusion model.
- Enhance → Sampling: samples · Enhancement invokes the reverse-process sampler.
- Sampling → Predictors: predict step · Sampling includes predictor updates.
- Sampling → Correctors: correct step · Sampling includes corrector updates.
- Metrics → Logs: reads outputs · Metrics are computed from generated artifacts and references.
- Model → Registry: configured by · Registry supports pluggable component selection by name.
- Inference utils → Tensor utils: uses · Inference helpers rely on tensor utilities.
- Inference utils → Other utils: uses · Inference helpers rely on general utilities.