How crlandsc/music-demixing-with-band-split-rnn works
Main Components
• Data Input & Preprocessing: This part involves reading the raw audio (wav/mp3) and converting them into spectrograms using Short-Time Fourier Transform (STFT). It also covers the dataset preparation (see prepare_dataset.py and the SAD (Source Activity Detection) module).
• Configuration & Utilities: The hydra YAML configuration files (in src/conf) control the training, data augmentation, callbacks, logger, and dataset splits.
• Model Architecture: Break the neural network into its three principal modules:
• Band-split Module: Converts audio to spectrogram and splits it (configured via src/conf/featurizer/stft.yaml and others).
• Band and Sequence Separation Module: Uses RNN components (typically bi-directional LSTMs) to process along time and frequency dimensions.
• Mask Estimation Module: Implements per-band MLPs to predict the spectrogram “mask” that’s later applied for audio separation.
• Training & Evaluation Pipelines: Represent components for training (train.py, PyTorch-Lightning module in src/model/pl_model.py), logging (callbacks and tensorboard/wandb configs inside src/conf/callbacks and src/conf/logger), and evaluation (evaluate.py).
• Inference Pipeline: A dedicated script (inference.py) that accepts input audio, invokes the model (with checkpoints stored in src/saved_models/*), and produces the separated stems.
• Notebooks: Used for EDA and data visualization, which gives insights into how the waveforms, spectra, and spectrograms look.
• External Dependencies: The dataset (MUSDB18 / Label Noise data), external libraries (PyTorch, PyTorch-Lightning, Hydra), and tools such as ffmpeg.
Relationships & Interactions
• Data Flow: Start with audio input → Preprocessing & feature extraction (STFT, normalization, splitting into bands) → Pass through the neural model (three modules: band-split → sequence separation → mask estimation) → Reconstruct separated audio via inverse transforms.
• Configuration Flow: Show hydra-driven configuration files from the src/conf directory not only affecting data augmentation, model behavior, and training parameters but also passed to both train.py and evaluate.py.
• Training Loop: Outline the training process including data ingestion, model forward pass, loss computation, callback logging, checkpoint saving (src/saved_models), and backpropagation.
• Evaluation & Inference: Connect the evaluation pipeline (which calculates uSDR and cSDR metrics) and the inference script where model checkpoints are loaded.
Architectural Patterns & Technologies
• Separation of Concerns: Clearly separate configuration (config YAML files), data processing (in src/data), model definitions (in src/model), and training/inference (in separate scripts).
• Modular Design: The model itself is split into distinct modules (using separate Python files for band-split and mask estimation), which allows easy swapping of, for example, target instrument-specific configurations.
• Use of PyTorch-Lightning and Hydra for streamlined training, logging, and experiment management.
• Utilization of external monitoring tools (TensorBoard, Weights & Biases) as indicated by configuration files in the logger directories.
Additional Details to Capture
• That different instruments (bass, drums, other, vocals) have dedicated configurations and checkpoints. It might be useful to show that the model training/inference pipeline can branch off into different target-specific models.
• Role played by utilities scripts (present in src/utils) in supporting callbacks, logging, and test utilities.
• That while the backend is a deep learning model, the “frontend” of this pipeline includes notebooks used for exploratory data analysis and presentation materials (in the presentation and images directories).