How rickyl-2000/rosvot works
Project type and purpose
• This is a research-grade ML codebase for “ROSVOT: Robust Singing Voice Transcription and MIDI Extraction.”
• It provides end-to-end data preprocessing, model training (including auxiliary models for pitch and word-boundary prediction), and inference pipelines for converting raw singing audio into aligned MIDI notes.
File structure
• Top-level directories:
• configs/ YAML hyperparameter and dataset configurations (rosvot, musan, rwbd, base)
• data_gen/ scripts to binarize audio datasets into model-readable format
• inference/ single-run and batched inference entrypoint
• modules/ model implementations (commons: conformer, transformer, rnn, wavenet; pe: RMVPE pitch extractor; rosvot: U-Net MIDI extractor)
• tasks/ training orchestration (dataset loading, training loops for rosvot/rwbd)
• utils/ I/O, audio processing, dataset, trainer, metrics, plotting, etc.
• scripts/ installation helper and M4Singer preprocessing
• Notice separation of concerns:
• Data pipeline (data_gen + utils/audio)
• Model library (modules/)
• Experiment orchestration (tasks/, configs/)
• Inference engine (inference/)
README for additional insights
• Dependencies: Python 3.9, PyTorch 2.1.1 (+ optional TensorFlow 2.9.0 for some utilities), CUDA 11.8
• Three core model weights:
1. ROSVOT (MIDI extraction)
2. RWBD (word boundary predictor)
3. RMVPE (pitch extractor)
• Bulk and single-run inference use the same core modules plus DDP for parallelism.
• Training relies on M4Singer + MUSAN noise dataset.