How fschmid56/pretrainedsed works
Project type and purpose
• This is a Python-based machine-learning codebase for pre-training, evaluating, and deploying audio spectrogram transformers on sound event detection tasks.
• Its primary goals are (a) to convert AudioSet MP3 data into a HF dataset, (b) to run a two-step pre-training pipeline (weak then strong labels), (c) to fine-tune on downstream tasks (DCASE 2016 Task 2), and (d) to provide an inference script.
File structure
• Top level:
• config.py: central hyperparameter and path definitions
• ex_audioset_strong.py & ex_dcase2016task2.py: training entrypoints
• inference.py: production inference
• requirements.txt, LICENSE, README.md
• data_util/: operations on AudioSet raw HDF5 files (class mappings, pseudo-label transforms)
• hf_dataset_gen/: scripts to build a Huggingface Dataset from AudioSet HDF5
• helpers/: shared augmentations, encoding/decoding, scoring utilities
• models/: one subfolder per transformer (ASiT, ATST-Frame, BEATs, fPaSST, M2D, frame_mn, frame_passt), each with:
– wrapper to load checkpoint and perform forward
– model definition and data transforms
– utility functions
• resources/: downloaded pre-trained checkpoints and ensemble logits
• test_files/: example audio for smoke-test
Additional README insights
• Two-stage pre-training:
Step 1: self-supervised + weak labels → produces “weak” checkpoints
Step 2: frame-level strong labels → yields “strong” checkpoints
• Sequence model heads (BiGRU, attention, transformer) are added atop pretrained backbones.
• Fine-tuning script ex_dcase2016task2.py shows linear vs full-model adaptation.
Main components
a. Data Ingestion & Preprocessing
• Raw AudioSet HDF5 files (balanced_train, unbalanced_train, eval)
• data_util (class label mapping, pseudo-labels)
• hf_dataset_gen → Huggingface dataset
• helpers/augment.py for online augmentations
b. Pre-training Pipeline
• Model wrappers in models/<model_name>
• Backbone Transformer (Vision-style spectrogram encoder)
• Sequence head (BiGRU/Attention/Transformer)
• Loss functions (CE for weak, PSDS/strong ranking for strong)
• ex_audioset_strong.py orchestrates training, checkpoint download/upload (resources), logging (wandb)
c. Evaluation
• helpers/score.py for PSDS metrics
• ex_audioset_strong.py --evaluate
d. Fine-tuning on downstream tasks
• ex_dcase2016task2.py
• Optionally freeze backbone, train only classification head
• HEAR dataset loader for DCASE 2016 Task 2
e. Inference
• inference.py loads resources (model checkpoints), accepts arbitrary audio, runs spectrogram transform → wrapper → post-processing decode.py
• Output sound event timestamps and scores
Architectural patterns & design principles
• Modular “wrapper” pattern: each transformer implementation is encapsulated under models/<model>
• Separation of concerns: data conversion (hf_dataset_gen), data utilities (data_util/helpers), model definitions (models), orchestration scripts (ex_*.py), inference
• Pluggable sequence heads: RNN/Transformer/Attention modules are interchangeable
• Environment-driven configuration: config.py + CLI args govern dataset paths, checkpoint sources, hyperparameters
Technologies & frameworks
• Python 3.9, PyTorch, Huggingface Datasets API
• Cython-based minimp3py for MP3 decoding
• HDF5 for AudioSet storage, CSV metadata
• WandB for experiment logging
• Optional GPU/CUDA integration
Essential elements
• Clear component labels