How bryanogya/multimodal-personalized-eq-prediction works
1. Repository purpose and end-to-end flow
- This is an offline ML research pipeline for predicting personalized EQ curves.
- The core flow is: raw audio/device/preference data → preprocessing and feature generation → multimodal training → checkpointed evaluation → visualizations and inference.
2. Configuration layer
- configs/ centralizes experiment parameters and file locations.
- It separates concerns by modality and purpose: audio, device, preferences, EQ frequency definitions, model shape, dataset splits, training hyperparameters, plot styling, and filesystem paths.
- This makes the rest of the code base largely config-driven rather than hardcoding paths or dimensions.
3. Data sources and repository-managed assets
- data/raw/audio, data/raw/device, and data/raw/harman contain source material for the three main signal types.
- data/metadata/ stores CSV metadata used to align samples, devices, and preference profiles.
- data/splits/ defines train/validation/test partitions, so dataset membership is explicit and reproducible.
4. Preprocessing subsystem
- src/preprocessing/ transforms raw inputs into model-ready artifacts.
- Audio preprocessing likely converts waveforms into compact spectral features such as mel-spectrograms.
- Device preprocessing normalizes or interpolates frequency response curves into a common EQ band representation.
- Harman preprocessing and utils/generate_target_eq.py indicate the project also builds target EQ curves from a reference response target.
5. Preference generation and personalization
- src/preprocessing/generate_preferences.py and configs/preferences.py define how user preference vectors are created or encoded.
- Preferences are treated as a separate modality rather than metadata-only, which is important architecturally: they feed the model as learnable or precomputed embeddings.
6. Dataset and loading boundary
- src/dataset/ is the boundary between stored artifacts and training-time tensors.
- data_loader.py and dataset.py package audio, device, and preference inputs together with target EQ labels.
- test_dataset.py suggests a special dataset path for inference or evaluation-only samples, keeping test-time access patterns separate from training.
7. Model architecture
- src/models/ contains separate encoders for audio, device, and preference streams, plus multimodal fusion models.
- multimodal_model.py is the main predictor; multimodal_model_aux.py indicates a secondary model for auxiliary response-curve prediction.
- The architecture is therefore modular: modality-specific encoders feed a fused predictor rather than one monolithic network.
8. Training subsystem
- src/training/ contains the training loops, validation logic, and loss definitions.
- train.py and validate.py handle the main multimodal model; train_aux.py and validate_aux.py handle the auxiliary model.
- losses.py suggests the optimization objective is tailored to EQ/curve regression rather than generic classification.
- Training outputs are persisted under checkpoints/, including histories and test metrics.
9. Evaluation and experiment comparison
- src/evaluation/ handles the post-training analysis path.
- baseline.py compares against simple reference strategies such as zero EQ and mean EQ.
- ablation.py isolates the contribution of each modality by running reduced-input variants.
- test.py, metrics.py, and worst_prediction.py imply a detailed test pass with aggregate scores plus error inspection.
10. Inference path
- src/inference/predict.py is the user-facing prediction entry point.
- It consumes processed audio, device, and preference artifacts and returns EQ predictions for a specific preference index.
- This is separate from training/evaluation so the inference boundary stays lightweight and does not depend on training-only code paths.
11. Utility and signal-processing helpers
- src/utils/ contains supporting functions for EQ interpolation, distribution checks, sanity checks, and experiment bookkeeping.
- These utilities bridge data-format mismatches, validate assumptions, and support reproducible experiment management.
- generate_target_eq.py and interpolate_eq.py are especially important for keeping all EQ curves aligned to the same frequency grid.
12. Visualization and reporting
- src/visualization/ turns experiment outputs into figures for training curves, prediction scatter plots, correlation matrices, histogram/error analysis, sensitivity plots, and ablation/baseline comparisons.
- outputs/figures/ is organized by analysis type, while checkpoints/*/*.json stores the numerical results that these plots summarize.
- This makes reporting a first-class subsystem rather than an ad hoc notebook step.
13. Experiment artifacts and lifecycle
- The checkpoints/ hierarchy records distinct experiment families: full_model, baseline, ablation, and aux_model.
- Each family stores histories, test metrics, and case studies like worst_predictions.json.
- The repo therefore supports repeatable experiment tracking across model variants, not just a single trained network.
14. Top-level orchestration
- main.py is the command-line dispatcher that selects training, auxiliary training, evaluation, baseline comparison, ablation, prediction, and plotting modes.
- Architecturally, it is the integration point that wires configs, datasets, models, training/evaluation code, and visualization outputs into a single workflow.
Main entry points: main.py main.py · dataset.py Dataset · data_loader.py Loader · multimodal_model.py Fusion model
Components 34
Config & data
Experiment settings, raw inputs, and split metadata.
- Audio cfg · Audio feature and preprocessing parameters.
configs/audio.py - Dataset cfg · Dataset split and sample-shaping parameters.
configs/dataset.py - EQ cfg · EQ target and curve-shaping settings.
configs/eq.py - Freq cfg · Frequency grid definitions used across the pipeline.
configs/frequencies.py - Model cfg · Architecture and hidden-size settings.
configs/model.py - Paths · Repository locations for data, checkpoints, and outputs.
configs/paths.py - Prefs cfg · Preference encoding and conditioning settings.
configs/preferences.py - Metadata CSVs · Audio, device, and preference alignment tables.
data/metadata - Splits · Train, validation, and test partition definitions.
data/splits - Raw sources · Audio, device responses, and Harman target source files.
data/raw
Prep & utils
Feature generation, alignment, and EQ helpers.
- Audio prep · Converts raw audio into model-ready features.
src/preprocessing/preprocess_audio.py - Device prep · Normalizes and interpolates device response curves.
src/preprocessing/preprocess_device.py - Harman prep · Builds target EQ curves from the Harman reference.
src/preprocessing/preprocess_harman.py - Prefs gen · Creates or encodes preference vectors.
src/preprocessing/generate_preferences.py - EQ utils · Interpolation, target generation, and EQ consistency checks.
src/utils/interpolate_eq.py - Experiment utils · Sanity checks and experiment bookkeeping helpers.
src/utils/experiment.py
Model & training
Dataset boundary, encoders, fusion model, and optimization.
- main.py · Dispatches training, evaluation, inference, and plotting workflows.
main.py - Dataset · Packages stored artifacts into training-time tensors.
src/dataset/dataset.py - Loader · Builds batch loading and sample collation.
src/dataset/data_loader.py - Test set · Inference/evaluation-only dataset path.
src/dataset/test_dataset.py - Audio encoder · Encodes audio-derived features.
src/models/audio_encoder.py - Device encoder · Encodes device response features.
src/models/device_encoder.py - Pref encoder · Encodes user preference vectors.
src/models/preference_encoder.py - Fusion model · Main multimodal EQ predictor.
src/models/multimodal_model.py - Aux model · Secondary response-curve prediction model.
src/models/multimodal_model_aux.py - Losses · Regression losses tailored to EQ prediction.
src/training/losses.py - Train · Main model training routine.
src/training/train.py - Validate · Main model validation routine.
src/training/validate.py - Train aux · Auxiliary model training routine.
src/training/train_aux.py - Validate aux · Auxiliary model validation routine.
src/training/validate_aux.py
Eval & reporting
Testing, baselines, ablations, inference, and plots.
- Test eval · Runs checkpointed test-set evaluation.
src/evaluation/test.py - Baseline · Compares against simple reference strategies.
src/evaluation/baseline.py - Ablation · Measures modality contribution via reduced-input runs.
src/evaluation/ablation.py - Metrics · Aggregates test scores and error statistics.
src/evaluation/metrics.py
Connections 42
- main.py → Train: run · Dispatches main training mode.
- main.py → Train aux: run · Dispatches auxiliary training mode.
- main.py → Test eval: run · Dispatches checkpoint evaluation.
- main.py → Baseline: run · Dispatches baseline comparison mode.
- main.py → Ablation: run · Dispatches ablation analysis mode.
- main.py → Baseline: predict · Dispatches prediction workflow.
- main.py → Experiment utils: plot · Dispatches reporting and experiment utilities.
- Paths → Raw sources: locates · Defines source-data locations.
- Paths → Metadata CSVs: locates · Defines metadata locations.
- Paths → Splits: locates · Defines split locations.
- Audio cfg → Audio prep: drives · Controls audio feature generation.
- EQ cfg → Harman prep: drives · Controls target EQ generation.
- Prefs cfg → Prefs gen: drives · Controls preference encoding.
- Freq cfg → EQ utils: defines grid · Shared frequency axis for interpolation and targets.
- Raw sources → Audio prep: input · Raw audio is transformed into features.
- Raw sources → Device prep: input · Raw device curves are normalized.
- Raw sources → Harman prep: input · Reference target data is processed.
- Metadata CSVs → Dataset: aligns · Metadata joins modalities and labels.
- Splits → Loader: partitions · Determines train/val/test membership.
- Audio prep → Dataset: feeds · Provides audio tensors or features.
- Device prep → Dataset: feeds · Provides device-response tensors.
- Prefs gen → Dataset: feeds · Provides preference vectors.
- Dataset → Loader: batching · Batched into training and evaluation samples.
- Dataset → Test set: specializes · Separate access path for inference/eval samples.
- Loader → Audio encoder: routes · Supplies audio modality inputs.
- Loader → Device encoder: routes · Supplies device modality inputs.
- Loader → Pref encoder: routes · Supplies preference modality inputs.
- Audio encoder → Fusion model: fuses into · Audio embedding enters the predictor.
- Device encoder → Fusion model: fuses into · Device embedding enters the predictor.
- Pref encoder → Fusion model: fuses into · Preference embedding enters the predictor.
- Fusion model → Losses: optimized by · Main predictor is trained with EQ regression losses.
- Aux model → Losses: optimized by · Auxiliary predictor uses the same regression family.
- Train → Fusion model: updates · Main loop trains the fusion model.
- Train → Validate: checks · Validation follows each training phase.
- Train → Losses: uses · Main optimization objective.
- Train aux → Aux model: updates · Auxiliary loop trains the secondary model.
- Train aux → Validate aux: checks · Validation for auxiliary model.
- Train aux → Losses: uses · Auxiliary optimization objective.
- Test eval → Metrics: computes · Produces aggregate test metrics.
- Test eval → Baseline: compares · Shares evaluation outputs with baseline analysis.
- Ablation → Metrics: computes · Produces ablation results.
- Baseline → Metrics: computes · Produces baseline metrics.