# rickyl-2000/rosvot architecture

> Architecture diagram of the GitHub repository rickyl-2000/rosvot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rickyl-2000/rosvot
- Repository: https://github.com/rickyl-2000/rosvot

## Overview

##### Project type and purpose
- This is a research-grade ML codebase for “ROSVOT: Robust Singing Voice Transcription and MIDI Extraction.”
- It provides end-to-end data preprocessing, model training (including auxiliary models for pitch and word-boundary prediction), and inference pipelines for converting raw singing audio into aligned MIDI notes.

##### File structure
- Top-level directories:
- configs/ YAML hyperparameter and dataset configurations (rosvot, musan, rwbd, base)
- data_gen/ scripts to binarize audio datasets into model-readable format
- inference/ single-run and batched inference entrypoint
- modules/ model implementations (commons: conformer, transformer, rnn, wavenet; pe: RMVPE pitch extractor; rosvot: U-Net MIDI extractor)
- tasks/ training orchestration (dataset loading, training loops for rosvot/rwbd)
- utils/ I/O, audio processing, dataset, trainer, metrics, plotting, etc.
- scripts/ installation helper and M4Singer preprocessing
- Notice separation of concerns:
- Data pipeline (data_gen + utils/audio)
- Model library (modules/)
- Experiment orchestration (tasks/, configs/)
- Inference engine (inference/)

##### README for additional insights
- Dependencies: Python 3.9, PyTorch 2.1.1 (+ optional TensorFlow 2.9.0 for some utilities), CUDA 11.8
- Three core model weights:
1. ROSVOT (MIDI extraction)
2. RWBD (word boundary predictor)
3. RMVPE (pitch extractor)
- Bulk and single-run inference use the same core modules plus DDP for parallelism.
- Training relies on M4Singer + MUSAN noise dataset.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources" 
        direction TB
        RawAudio["Raw Singing Audio\n(M4Singer/User)"]:::ext
        MUSAN["MUSAN Noise Source"]:::ext
        MFA["Forced-Aligner Metadata"]:::ext
    end

    %% Preprocessing Pipeline
    subgraph "Preprocessing Pipeline" 
        direction TB
        DGRun["data_gen/run.py"]:::dataBox
        BaseBin["base_binarizer.py"]:::dataBox
        MusanBin["musan_binarizer.py"]:::dataBox
        RosvotBin["rosvot_binarizer.py"]:::dataBox
        PrepM4["scripts/prepare_m4singer.py"]:::dataBox
    end

    BinData[(Binary Dataset)]:::store

    %% Data Loader & Augmentation
    subgraph "Data Loader & Augmentation"
        direction TB
        DatasetUtils["dataset_utils.py"]:::utilBox
        MultiProc["multiprocess_utils.py"]:::utilBox
        IO["io.py"]:::utilBox
        Mel["mel.py"]:::utilBox
        PitchUtils["pitch_utils.py"]:::utilBox
        VAD["vad.py"]:::utilBox
        Align["align.py"]:::utilBox
    end

    %% Model Libraries
    subgraph "Model Libraries"
        direction TB
        subgraph "RMVPE Pitch Extractor" 
            direction TB
            RMConst["constants.py"]:::modelBox
            RMDeep["deepunet.py"]:::modelBox
            RMExtr["extractor.py"]:::modelBox
            RMInf["inference.py"]:::modelBox
            RMModel["model.py"]:::modelBox
            RMSeq["seq.py"]:::modelBox
            RMSpec["spec.py"]:::modelBox
            RMUtils["utils.py"]:::modelBox
        end
        subgraph "RWBD Word-Boundary Predictor"
            RosvotUtils["rosvot_utils.py"]:::modelBox
            CommonLayers["commons Layers"]:::modelBox
        end
        subgraph "ROSVOT MIDI Extractor"
            RosvotCore["rosvot.py"]:::modelBox
            RosvotUNet["unet.py"]:::modelBox
        end
        subgraph "Shared Building Blocks"
            Conformer["conformer.py"]:::modelBox
            ConfLayers["layers.py"]:::modelBox
            PosEmbed["espnet_positional_embedding.py"]:::modelBox
            Attn["espnet_transformer_attn.py"]:::modelBox
            Conv["conv.py"]:::modelBox
            Transformer["transformer.py"]:::modelBox
            RNN["rnn.py"]:::modelBox
            Wavenet["wavenet.py"]:::modelBox
            RelTrans["rel_transformer.py"]:::modelBox
            CommLayers["layers.py"]:::modelBox
        end
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        direction TB
        RunTask["tasks/run.py"]:::orchBox
        RosvotTask["tasks/rosvot/task.py"]:::orchBox
        Trainer["trainer.py"]:::utilBox
        DDPUtils["ddp_utils.py"]:::utilBox
        CKPTUtils["ckpt_utils.py"]:::utilBox
    end

    Checkpoints[(Checkpoints Storage)]:::store

    %% Inference Engine
    subgraph "Inference Engine"
        direction TB
        InferRos["inference/rosvot.py"]:::orchBox
    end

    %% Utilities & Metrics
    subgraph "Utilities & Metrics"
        direction TB
        DTW["dtw.py"]:::utilBox
        PitchDist["pitch_distance.py"]:::utilBox
        SSIM["ssim.py"]:::utilBox
        DiagMet["diagonal_metrics.py"]:::utilBox
        LapVar["laplace_var.py"]:::utilBox
        Plot["plot.py"]:::utilBox
    end

    %% Relationships
    RawAudio -->|feeds| DGRun
    MUSAN -->|feeds| MusanBin
    PrepM4 -->|generates metadata| DGRun
    DGRun -->|uses| BaseBin
    DGRun -->|uses| MusanBin
    DGRun -->|uses| RosvotBin
    DGRun -->|outputs| BinData
    BinData -->|loaded by| DatasetUtils
    DatasetUtils -->|calls| MultiProc
    DatasetUtils -->|calls| IO
    IO --> Mel
    IO --> PitchUtils
    IO --> VAD
    IO --> Align
    DatasetUtils -->|batches| BinData

    BinData --> RMInf
    RMInf --> RMModel
    RMModel --> RMDeep
    RMDeep --> RMConst
    RMModel --> RMSeq
    RMModel --> RMSpec
    RMModel --> RMUtils

    BinData --> RosvotUtils
    RosvotUtils --> CommonLayers

    BinData --> RosvotCore
    RosvotCore --> RosvotUNet
    RosvotCore --> CommonLayers

    CommonLayers --> Conformer
    CommonLayers --> Conv
    CommonLayers --> Transformer
    CommonLayers --> RNN
    CommonLayers --> Wavenet
    Conformer --> PosEmbed
    Conformer --> Attn
    Transformer --> RelTrans
    Transformer --> CommLayers

    %% Training connections
    DatasetUtils -->|provides tensors| RunTask
    RunTask -->|trains| RosvotTask
    RunTask -->|uses| Trainer
    Trainer -->|uses| DDPUtils
    Trainer -->|uses| CKPTUtils
    Trainer -->|writes| Checkpoints

    %% Inference connections
    RawAudio -->|single/batch| InferRos
    InferRos --> RMInf
    InferRos --> RosvotUtils
    InferRos --> RosvotCore
    InferRos -->|outputs| Plot

    %% Metrics
    InferRos --> DTW
    InferRos --> PitchDist
    InferRos --> SSIM
    InferRos --> DiagMet
    InferRos --> LapVar

    %% Click Events
    click DGRun "https://github.com/rickyl-2000/rosvot/blob/main/data_gen/run.py"
    click BaseBin "https://github.com/rickyl-2000/rosvot/blob/main/data_gen/base_binarizer.py"
    click MusanBin "https://github.com/rickyl-2000/rosvot/blob/main/data_gen/musan_binarizer.py"
    click RosvotBin "https://github.com/rickyl-2000/rosvot/blob/main/data_gen/rosvot_binarizer.py"
    click PrepM4 "https://github.com/rickyl-2000/rosvot/blob/main/scripts/prepare_m4singer.py"
    click DatasetUtils "https://github.com/rickyl-2000/rosvot/blob/main/utils/commons/dataset_utils.py"
    click MultiProc "https://github.com/rickyl-2000/rosvot/blob/main/utils/commons/multiprocess_utils.py"
    click IO "https://github.com/rickyl-2000/rosvot/blob/main/utils/audio/io.py"
    click Mel "https://github.com/rickyl-2000/rosvot/blob/main/utils/audio/mel.py"
    click PitchUtils "https://github.com/rickyl-2000/rosvot/blob/main/utils/audio/pitch_utils.py"
    click VAD "https://github.com/rickyl-2000/rosvot/blob/main/utils/audio/vad.py"
    click Align "https://github.com/rickyl-2000/rosvot/blob/main/utils/audio/align.py"
    click RMConst "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/constants.py"
    click RMDeep "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/deepunet.py"
    click RMExtr "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/extractor.py"
    click RMInf "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/inference.py"
    click RMModel "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/model.py"
    click RMSeq "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/seq.py"
    click RMSpec "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/spec.py"
    click RMUtils "https://github.com/rickyl-2000/rosvot/blob/main/modules/pe/rmvpe/utils.py"
    click RosvotUtils "https://github.com/rickyl-2000/rosvot/blob/main/tasks/rosvot/rosvot_utils.py"
    click RosvotCore "https://github.com/rickyl-2000/rosvot/blob/main/modules/rosvot/rosvot.py"
    click RosvotUNet "https://github.com/rickyl-2000/rosvot/blob/main/modules/rosvot/unet.py"
    click Conformer "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/conformer/conformer.py"
    click ConfLayers "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/conformer/layers.py"
    click PosEmbed "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/conformer/espnet_positional_embedding.py"
    click Attn "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/conformer/espnet_transformer_attn.py"
    click Conv "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/conv.py"
    click Transformer "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/transformer.py"
    click RNN "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/rnn.py"
    click Wavenet "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/wavenet.py"
    click RelTrans "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/rel_transformer.py"
    click CommLayers "https://github.com/rickyl-2000/rosvot/blob/main/modules/commons/layers.py"
    click RunTask "https://github.com/rickyl-2000/rosvot/blob/main/tasks/run.py"
    click RosvotTask "https://github.com/rickyl-2000/rosvot/blob/main/tasks/rosvot/task.py"
    click Trainer "https://github.com/rickyl-2000/rosvot/blob/main/utils/commons/trainer.py"
    click DDPUtils "https://github.com/rickyl-2000/rosvot/blob/main/utils/commons/ddp_utils.py"
    click CKPTUtils "https://github.com/rickyl-2000/rosvot/blob/main/utils/commons/ckpt_utils.py"
    click InferRos "https://github.com/rickyl-2000/rosvot/blob/main/inference/rosvot.py"
    click DTW "https://github.com/rickyl-2000/rosvot/blob/main/utils/metrics/dtw.py"
    click PitchDist "https://github.com/rickyl-2000/rosvot/blob/main/utils/metrics/pitch_distance.py"
    click SSIM "https://github.com/rickyl-2000/rosvot/blob/main/utils/metrics/ssim.py"
    click DiagMet "https://github.com/rickyl-2000/rosvot/blob/main/utils/metrics/diagonal_metrics.py"
    click LapVar "https://github.com/rickyl-2000/rosvot/blob/main/utils/metrics/laplace_var.py"
    click Plot "https://github.com/rickyl-2000/rosvot/blob/main/utils/plot/plot.py"

    %% Styles
    classDef dataBox fill:#BBDEFB,stroke:#0D47A1,color:#0D47A1
    classDef modelBox fill:#C8E6C9,stroke:#2E7D32,color:#2E7D32
    classDef orchBox fill:#FFE0B2,stroke:#EF6C00,color:#E65100
    classDef utilBox fill:#E0E0E0,stroke:#424242,color:#424242
    classDef store fill:#FFF9C4,stroke:#F57F17,color:#F57F17
    classDef ext fill:#E1BEE7,stroke:#6A1B9A,color:#6A1B9A
```
