# crlandsc/music-demixing-with-band-split-rnn architecture

> Architecture diagram of the GitHub repository crlandsc/music-demixing-with-band-split-rnn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crlandsc/music-demixing-with-band-split-rnn
- Repository: https://github.com/crlandsc/music-demixing-with-band-split-rnn

## Overview

##### Main Components
- Data Input & Preprocessing: This part involves reading the raw audio (wav/mp3) and converting them into spectrograms using Short-Time Fourier Transform (STFT). It also covers the dataset preparation (see prepare_dataset.py and the SAD (Source Activity Detection) module).
- Configuration & Utilities: The hydra YAML configuration files (in src/conf) control the training, data augmentation, callbacks, logger, and dataset splits.
- Model Architecture: Break the neural network into its three principal modules:
- Band-split Module: Converts audio to spectrogram and splits it (configured via src/conf/featurizer/stft.yaml and others).
- Band and Sequence Separation Module: Uses RNN components (typically bi-directional LSTMs) to process along time and frequency dimensions.
- Mask Estimation Module: Implements per-band MLPs to predict the spectrogram “mask” that’s later applied for audio separation.
- Training & Evaluation Pipelines: Represent components for training (train.py, PyTorch-Lightning module in src/model/pl_model.py), logging (callbacks and tensorboard/wandb configs inside src/conf/callbacks and src/conf/logger), and evaluation (evaluate.py).
- Inference Pipeline: A dedicated script (inference.py) that accepts input audio, invokes the model (with checkpoints stored in src/saved_models/*), and produces the separated stems.
- Notebooks: Used for EDA and data visualization, which gives insights into how the waveforms, spectra, and spectrograms look.
- External Dependencies: The dataset (MUSDB18 / Label Noise data), external libraries (PyTorch, PyTorch-Lightning, Hydra), and tools such as ffmpeg.

##### Relationships & Interactions
- Data Flow: Start with audio input → Preprocessing & feature extraction (STFT, normalization, splitting into bands) → Pass through the neural model (three modules: band-split → sequence separation → mask estimation) → Reconstruct separated audio via inverse transforms.
- Configuration Flow: Show hydra-driven configuration files from the src/conf directory not only affecting data augmentation, model behavior, and training parameters but also passed to both train.py and evaluate.py.
- Training Loop: Outline the training process including data ingestion, model forward pass, loss computation, callback logging, checkpoint saving (src/saved_models), and backpropagation.
- Evaluation & Inference: Connect the evaluation pipeline (which calculates uSDR and cSDR metrics) and the inference script where model checkpoints are loaded.

##### Architectural Patterns & Technologies
- Separation of Concerns: Clearly separate configuration (config YAML files), data processing (in src/data), model definitions (in src/model), and training/inference (in separate scripts).
- Modular Design: The model itself is split into distinct modules (using separate Python files for band-split and mask estimation), which allows easy swapping of, for example, target instrument-specific configurations.
- Use of PyTorch-Lightning and Hydra for streamlined training, logging, and experiment management.
- Utilization of external monitoring tools (TensorBoard, Weights & Biases) as indicated by configuration files in the logger directories.

##### Additional Details to Capture
- That different instruments (bass, drums, other, vocals) have dedicated configurations and checkpoints. It might be useful to show that the model training/inference pipeline can branch off into different target-specific models.
- Role played by utilities scripts (present in src/utils) in supporting callbacks, logging, and test utilities.
- That while the backend is a deep learning model, the “frontend” of this pipeline includes notebooks used for exploratory data analysis and presentation materials (in the presentation and images directories).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Input & Preprocessing
    subgraph "Data Input & Preprocessing"
        DI1["Audio Input"]:::data
        DI2["Preprocessing Module"]:::data
        DI3["Dataset Preparation"]:::data
        DI4["SAD Config"]:::data
    end

    %% Configuration & Utilities
    subgraph "Configuration & Utilities"
        C1["Hydra Config"]:::config
        C2["STFT Config"]:::config
        C3["Other YAML Configs"]:::config
    end

    %% Model Architecture
    subgraph "Model Architecture"
        MA2["Band-Split Config"]:::model
        MA1["Band-Split Module"]:::model
        MA3["RNN Sequence Module"]:::model
        MA4["Mask Estimation Module"]:::model
        MA5["Reconstructed Audio"]:::model
    end

    %% Training & Evaluation Pipeline
    subgraph "Training & Evaluation"
        TE1["Training Script"]:::training
        TE2["PL Model"]:::training
        TE3["Evaluation Script"]:::training
        TE4["Callbacks Config"]:::training
        TE5["Logger Config"]:::training
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        IP1["Inference Script"]:::inference
        IP2["Checkpoint Storage"]:::inference
    end

    %% Notebooks for EDA and Visualization
    subgraph "Notebooks"
        NB1["EDA Notebooks"]:::notebook
    end

    %% Utilities
    subgraph "Utilities"
        U1["Utils Scripts"]:::util
    end

    %% Connections for Data Flow
    DI1 --> DI2
    DI2 --> DI3
    DI3 --> DI4
    DI2 -->|"produces_features"| MA1

    %% Connections within Model Architecture
    MA2 --> MA1
    MA1 -->|"processes"| MA3
    MA3 -->|"predicts_mask"| MA4
    MA4 -->|"applies_inverse_STFT"| MA5

    %% Connections for Configuration Flow
    C2 --> DI2
    C1 -->|"configures"| TE1
    C1 -->|"configures"| IP1

    %% Connections for Training & Evaluation Pipeline
    TE1 --> TE2
    TE2 --> TE3
    TE2 --> TE4
    TE2 --> TE5

    %% Connections for Inference Pipeline
    IP1 --> IP2

    %% Additional Utility Connections
    U1 --> TE4

    %% Class Definitions (Color Styles)
    classDef data fill:#E0F7FA,stroke:#006064,stroke-width:2px;
    classDef config fill:#FFF3E0,stroke:#E65100,stroke-width:2px;
    classDef model fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px;
    classDef training fill:#F3E5F5,stroke:#6A1B9A,stroke-width:2px;
    classDef inference fill:#E3F2FD,stroke:#1565C0,stroke-width:2px;
    classDef notebook fill:#FFFDE7,stroke:#F57F17,stroke-width:2px;
    classDef util fill:#FBE9E7,stroke:#BF360C,stroke-width:2px;

    %% Click Events for Data Input & Preprocessing
    click DI2 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/data/preprocessing.py"
    click DI3 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/prepare_dataset.py"
    click DI4 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/conf/sad/default.yaml"

    %% Click Events for Configuration & Utilities
    click C1 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/conf/config.yaml"
    click C2 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/conf/featurizer/stft.yaml"

    %% Click Events for Model Architecture
    click MA1 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/model/modules/bandsplit.py"
    click MA2 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/conf/model/bandsplitrnnV7.yaml"
    click MA3 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/model/modules/bandsequence.py"
    click MA4 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/model/modules/maskestimation.py"

    %% Click Events for Training & Evaluation Pipeline
    click TE1 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/train.py"
    click TE2 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/model/pl_model.py"
    click TE3 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/evaluate.py"
    click TE4 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/tree/main/src/conf/callbacks"
    click TE5 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/tree/main/src/conf/logger"

    %% Click Events for Inference Pipeline
    click IP1 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/blob/main/src/inference.py"
    click IP2 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/tree/main/src/saved_models"

    %% Click Events for Notebooks and Utilities
    click NB1 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/tree/main/src/notebooks"
    click U1 "https://github.com/crlandsc/music-demixing-with-band-split-rnn/tree/main/src/utils"
```
