How lrenault/ddsp-piano works
Project Type and Purpose
• DDSP-Piano is an open-source research tool (CLI-based Python package) for MIDI-to-audio performance synthesis. It leverages the DDSP TensorFlow library to build a differentiable piano model, train it on the MAESTRO dataset, evaluate it, and run inference (synthesis) from MIDI or CSV inputs.
File Structure
• Top-level scripts (CLI entry points):
• preprocess_maestro.py – convert MAESTRO MIDI+audio into TFRecords
• train_single_phase.py & train_ddsp_piano.sh – train the model
• evaluate_model.py – evaluate on the MAESTRO test set
• synthesize_midi_file.py & synthesize_from_csv.py – run inference
• ddsp_piano/ package:
• configs/ – .gin model configurations
• data_pipeline.py – reads TFRecords, batches data
• default_model.py – assembles the TF model from modules
• modules/ – core synthesis building blocks (oscillators, reverb, losses)
• utils/ – I/O helpers, MIDI encoders, logging summaries
• model_weights/ – pre‐trained checkpoints
• Training outputs: experiment directories with logs/, checkpoints, TensorBoard data
Extract Additional Insights from the README
Data flow → Encoder (MIDI features) → Core DDSP synthesis modules → Reverb → Audio
• Dependencies: TensorFlow 2.x, DDSP library, Gin-Config
• Major operations: preprocessing, single-phase training, multi-phase (legacy), batch & single inference, evaluation
Main Components & Their Interactions
a. Components
1. Data Sources
– Raw MAESTRO dataset (MIDI + WAV)
2. Preprocessing
– preprocess_maestro.py → TFRecords storage
3. Configuration Manager
– .gin files in configs/ determine model hyperparameters and architecture
4. Data Pipeline
– data_pipeline.py loads TFRecords, applies on-the-fly MIDI encoding / audio resampling
5. Model Builder
– default_model.py + modules/ assemble the DDSP graph (partials, oscillators, reverb)
6. Training Engine
– train_single_phase.py & train_ddsp_piano.sh
– Inputs: Data Pipeline + Model Builder + configs + training hyperparameters
– Outputs: Checkpoints → ddsp_piano/model_weights or <experiment>/phase_X/
– Logs: TensorBoard under <experiment>/logs/
7. Inference Engine
– synthesize_midi_file.py / synthesize_from_csv.py
– Inputs: MIDI file or CSV batch, config, checkpoint
– Processing: Load Model Builder + weights, run forward pass
– Outputs: WAV files (with optional dry vs. reverbed tracks)
8. Evaluation Engine
– evaluate_model.py
– Inputs: MAESTRO TFRecords, config, checkpoint
– Outputs: Objective metrics, optional audio samples
9. External Dependencies
– TensorFlow / DDSP library
– Gin-Config for hyperparameter injection
b. Relationships & Data Flows
c. Architectural Patterns & Principles
– Modular separation (data / config / model / utils)
– Pipeline pattern for data ingestion → training → inference
– Dependency injection via Gin-Config
– CLI orchestration for each major phase (preprocess, train, eval, synthesize)
d. Key Technologies & Frameworks
– Python 3
– TensorFlow 2.x & DDSP library
– Gin-Config for hyperparameters
– TFRecord format for fast I/O
– Bash scripts for full training orchestration
– CSV/MIDI input handling
– TensorBoard for logging and visualizations
Detail
• Config files as first-class components that feed into model construction.
• Two inference modes (single MIDI vs. batch CSV) converging on the same Inference Engine.
• Don’t omit the evaluation path (metrics + optional audio).