# fschmid56/pretrainedsed architecture

> Architecture diagram of the GitHub repository fschmid56/pretrainedsed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fschmid56/pretrainedsed
- Repository: https://github.com/fschmid56/pretrainedsed

## Overview

##### Project type and purpose
- This is a Python-based machine-learning codebase for pre-training, evaluating, and deploying audio spectrogram transformers on sound event detection tasks.
- Its primary goals are (a) to convert AudioSet MP3 data into a HF dataset, (b) to run a two-step pre-training pipeline (weak then strong labels), (c) to fine-tune on downstream tasks (DCASE 2016 Task 2), and (d) to provide an inference script.

##### File structure
- Top level:
- config.py: central hyperparameter and path definitions
- ex_audioset_strong.py & ex_dcase2016task2.py: training entrypoints
- inference.py: production inference
- requirements.txt, LICENSE, README.md
- data_util/: operations on AudioSet raw HDF5 files (class mappings, pseudo-label transforms)
- hf_dataset_gen/: scripts to build a Huggingface Dataset from AudioSet HDF5
- helpers/: shared augmentations, encoding/decoding, scoring utilities
- models/: one subfolder per transformer (ASiT, ATST-Frame, BEATs, fPaSST, M2D, frame_mn, frame_passt), each with:
– wrapper to load checkpoint and perform forward
– model definition and data transforms
– utility functions
- resources/: downloaded pre-trained checkpoints and ensemble logits
- test_files/: example audio for smoke-test

##### Additional README insights
- Two-stage pre-training:
Step 1: self-supervised + weak labels → produces “weak” checkpoints
Step 2: frame-level strong labels → yields “strong” checkpoints
- Sequence model heads (BiGRU, attention, transformer) are added atop pretrained backbones.
- Fine-tuning script ex_dcase2016task2.py shows linear vs full-model adaptation.

##### Main components
a. Data Ingestion & Preprocessing
- Raw AudioSet HDF5 files (balanced_train, unbalanced_train, eval)
- data_util (class label mapping, pseudo-labels)
- hf_dataset_gen → Huggingface dataset
- helpers/augment.py for online augmentations
b. Pre-training Pipeline
- Model wrappers in models/<model_name>
- Backbone Transformer (Vision-style spectrogram encoder)
- Sequence head (BiGRU/Attention/Transformer)
- Loss functions (CE for weak, PSDS/strong ranking for strong)
- ex_audioset_strong.py orchestrates training, checkpoint download/upload (resources), logging (wandb)
c. Evaluation
- helpers/score.py for PSDS metrics
- ex_audioset_strong.py --evaluate
d. Fine-tuning on downstream tasks
- ex_dcase2016task2.py
- Optionally freeze backbone, train only classification head
- HEAR dataset loader for DCASE 2016 Task 2
e. Inference
- inference.py loads resources (model checkpoints), accepts arbitrary audio, runs spectrogram transform → wrapper → post-processing decode.py
- Output sound event timestamps and scores

##### Architectural patterns & design principles
- Modular “wrapper” pattern: each transformer implementation is encapsulated under models/<model>
- Separation of concerns: data conversion (hf_dataset_gen), data utilities (data_util/helpers), model definitions (models), orchestration scripts (ex_*.py), inference
- Pluggable sequence heads: RNN/Transformer/Attention modules are interchangeable
- Environment-driven configuration: config.py + CLI args govern dataset paths, checkpoint sources, hyperparameters

##### Technologies & frameworks
- Python 3.9, PyTorch, Huggingface Datasets API
- Cython-based minimp3py for MP3 decoding
- HDF5 for AudioSet storage, CSV metadata
- WandB for experiment logging
- Optional GPU/CUDA integration

##### Essential elements
- Clear component labels

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        HDF5["AudioSet HDF5"]:::data
        cls_map["Class‐label mapping"]:::data
        pseudo["Pseudo‐label transforms"]:::data
        dcase_loader["DCASE2016 Loader"]:::data
        transforms["Audio Transforms"]:::data
    end

    %% HF Dataset Generation
    subgraph "HF Dataset Generation"
        hf_gen["hf_dataset_gen"]:::data
        meta1["class_labels_indices.csv"]:::data
        meta2["class_labels_indices_strong.csv"]:::data
        meta3["audioset_train_strong.csv"]:::data
        meta4["audioset_eval_strong.csv"]:::data
        HF_DS["Huggingface Datasets"]:::data
    end

    %% Helpers & Utilities
    subgraph "Helpers & Utilities"
        augment["Augmentations"]:::util
        encode["Encoding Utilities"]:::util
        decode["Decoding Utilities"]:::util
        score["Scoring/PSDS"]:::util
        utils["Misc Helpers"]:::util
    end

    %% Configuration & Resources
    subgraph "Configuration & Resources"
        config["config.py"]:::config
        resources["resources/"]:::config
    end

    %% Pre-training Orchestration
    ex_train["ex_audioset_strong.py"]:::script

    %% Transformer Backbones & Wrappers
    subgraph "Transformer Backbones & Wrappers"
        ASiT["ASiT Wrapper"]:::model
        ATSTF["ATST-Frame Wrapper"]:::model
        BEATs["BEATs Wrapper"]:::model
        FrameMN["Frame-MN Wrapper"]:::model
        FramePaSST["Frame-PaSST Wrapper"]:::model
        M2D["M2D Wrapper"]:::model
    end

    %% Sequence Heads
    seq_heads["Sequence Heads"]:::model

    %% Fine-tuning
    ex_ft["ex_dcase2016task2.py"]:::script

    %% Inference
    subgraph "Inference"
        inference["inference.py"]:::script
        output["Event Timestamps & Scores"]:::data
    end

    %% Relationships
    HDF5 --> cls_map
    HDF5 --> pseudo
    HDF5 --> dcase_loader
    HDF5 --> transforms

    cls_map --> hf_gen
    pseudo --> hf_gen
    dcase_loader --> hf_gen
    transforms --> hf_gen

    hf_gen --> meta1
    hf_gen --> meta2
    hf_gen --> meta3
    hf_gen --> meta4
    hf_gen --> HF_DS

    HF_DS --> ex_train
    augment --> ex_train
    encode --> ex_train
    decode --> ex_train
    score --> ex_train
    utils --> ex_train
    config --> ex_train
    resources --> ex_train

    ex_train --> ASiT
    ex_train --> ATSTF
    ex_train --> BEATs
    ex_train --> FrameMN
    ex_train --> FramePaSST
    ex_train --> M2D

    ASiT --> seq_heads
    ATSTF --> seq_heads
    BEATs --> seq_heads
    FrameMN --> seq_heads
    FramePaSST --> seq_heads
    M2D --> seq_heads

    seq_heads --> ex_train

    ex_train --> score

    ex_train -.-> resources
    resources --> ex_ft
    dcase_loader --> ex_ft

    ex_ft --> resources

    inference --> output
    resources --> inference
    decode --> inference
    ex_ft --> inference

    %% Click Events
    click cls_map "https://github.com/fschmid56/pretrainedsed/blob/main/data_util/audioset_classes.py"
    click pseudo "https://github.com/fschmid56/pretrainedsed/blob/main/data_util/audioset_strong.py"
    click dcase_loader "https://github.com/fschmid56/pretrainedsed/blob/main/data_util/dcase2016task2.py"
    click transforms "https://github.com/fschmid56/pretrainedsed/blob/main/data_util/transforms.py"
    click hf_gen "https://github.com/fschmid56/pretrainedsed/blob/main/hf_dataset_gen/audioset_strong.py"
    click meta1 "https://github.com/fschmid56/pretrainedsed/blob/main/hf_dataset_gen/metadata/class_labels_indices.csv"
    click meta2 "https://github.com/fschmid56/pretrainedsed/blob/main/hf_dataset_gen/metadata/class_labels_indices_strong.csv"
    click meta3 "https://github.com/fschmid56/pretrainedsed/blob/main/hf_dataset_gen/metadata/audioset_train_strong.csv"
    click meta4 "https://github.com/fschmid56/pretrainedsed/blob/main/hf_dataset_gen/metadata/audioset_eval_strong.csv"
    click augment "https://github.com/fschmid56/pretrainedsed/blob/main/helpers/augment.py"
    click encode "https://github.com/fschmid56/pretrainedsed/blob/main/helpers/encode.py"
    click decode "https://github.com/fschmid56/pretrainedsed/blob/main/helpers/decode.py"
    click score "https://github.com/fschmid56/pretrainedsed/blob/main/helpers/score.py"
    click utils "https://github.com/fschmid56/pretrainedsed/blob/main/helpers/utils.py"
    click config "https://github.com/fschmid56/pretrainedsed/blob/main/config.py"
    click resources "https://github.com/fschmid56/pretrainedsed/tree/main/resources/"
    click ex_train "https://github.com/fschmid56/pretrainedsed/blob/main/ex_audioset_strong.py"
    click ASiT "https://github.com/fschmid56/pretrainedsed/blob/main/models/asit/ASIT_wrapper.py"
    click ATSTF "https://github.com/fschmid56/pretrainedsed/blob/main/models/atstframe/ATSTF_wrapper.py"
    click BEATs "https://github.com/fschmid56/pretrainedsed/blob/main/models/beats/BEATs_wrapper.py"
    click FrameMN "https://github.com/fschmid56/pretrainedsed/blob/main/models/frame_mn/Frame_MN_wrapper.py"
    click FramePaSST "https://github.com/fschmid56/pretrainedsed/blob/main/models/frame_passt/fpasst_wrapper.py"
    click M2D "https://github.com/fschmid56/pretrainedsed/blob/main/models/m2d/M2D_wrapper.py"
    click seq_heads "https://github.com/fschmid56/pretrainedsed/blob/main/models/seq_models.py"
    click ex_ft "https://github.com/fschmid56/pretrainedsed/blob/main/ex_dcase2016task2.py"
    click inference "https://github.com/fschmid56/pretrainedsed/blob/main/inference.py"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72;
    classDef model fill:#ABEBC6,stroke:#145A32;
    classDef script fill:#FAD7A0,stroke:#B9770E;
    classDef config fill:#D7DBDD,stroke:#424949;
    classDef util fill:#E8DAEF,stroke:#6C3483;
```
