How sonyresearch/diffvox works
Project type and purpose
• This is a research‐oriented, full‐pipeline Python application for training and analysing a differentiable vocal effects model (“DiffVox”).
• It covers data ingestion, model definition, training (parameter retrieval), preset collection, evaluation, PCA analysis, and visualization.
File structure
• Top level:
• cfg/ – configuration YAMLs (Hydra) and model definitions
• data/ – data‐loading logic (MedleyDB, internal datasets)
• loss/ – loss functions and evaluation script (loss_eval.py)
• modules/ – core differentiable DSP modules (functional, fx, real‐time, utilities)
• presets/ – precomputed parameter sets and PCA artifacts for two datasets
• scripts/ – auxiliary shell and Python scripts for dataset prep, parameter collection, PCA
• main.py – entry point for training/retrieval
• visualisation.ipynb – notebook for figures in the paper
• Build/config: requirements.txt, .gitignore, LICENSE, README.md
README for additional insights
• Uses Hydra for flexible configuration (cfg/config.yaml)
• Supports single‐track retrieval and batch MedleyDB processing (train_medley.sh)
• Parameter collection (get_params.py), evaluation (loss_eval.py), PCA transform (pca_trsfm.py)
• Generates presets in numpy/json format, then analyzes with notebooks or external tools
• Configuration Manager (Hydra, cfg/)
• Data Loader (data/)
• Model & DSP Chain (modules/functional.py, fx.py, rt.py)
• Loss & Evaluation (loss/, loss_eval.py)
• Training Orchestrator (main.py)
• Parameter Collector & Preset Manager (scripts/get_params.py, presets/)
• PCA Analyzer (scripts/pca_trsfm.py)
• Visualization Layer (visualisation.ipynb)
• External Dataset (MedleyDB, internal audio files)
• Storage/File System (logs, presets folder, output CSVs/NPYs)
1. Configuration Manager → Training Orchestrator
2. Training Orchestrator ↔ Data Loader (fetch audio)
3. Training Orchestrator → Model & DSP Chain → Loss & Evaluation → logs/checkpoints
4. Logs → Parameter Collector → presets/
5. Presets → PCA Analyzer → updated presets/
6. Loss/Evaluation → CSV outputs → Visualization Layer
7. External Dataset → Data Loader
c. Architectural patterns & design principles:
• Modular separation: data, model, loss, scripts
• Config-driven pipelines (Hydra) for reproducibility
• Batch vs single‐track modes via scripts vs main entry point
• Reuse of numpy artifacts for cross‐step handoff (raw_params.npy, feature_mask.npy, gaussian.npz)
• Optional GPU vs CPU evaluation (flag-driven)
d. Key technologies/frameworks:
• Python 3.x, numpy, scipy
• Hydra for configuration
• Numba for real‐time DSP modules
• YAML for configs, JSON for metadata
• Bash scripts for batch processing
• Jupyter for visualization
• Optional GPU (PyTorch or custom autodiff – inferred)