How misya11p/amt-apc works
Project type and purpose
• This is an open-source ML pipeline for automatic piano cover generation (AMT-APC).
• It comprises data ingestion & preparation, model fine-tuning/training, inference (cover generation), and evaluation.
• Key use cases: prepare a dataset from raw audio/MIDI, fine-tune a pre-trained AMT model, generate piano covers, and compute evaluation metrics.
File structure
Top-level directories:
• data – scripts for downloading raw audio, syncing with MIDI, extracting features (style vectors), labeling, and assembling the PyTorch dataset.
• dataset – metadata (src.json) output from data scripts.
• infer – CLI entry point for inference (cover generation) using the fine-tuned model; supports WAV files or YouTube URLs and style options.
• train – CLI for training/fine-tuning (__main__.py, trainer, loss).
• eval – scripts implementing cover-based metrics (cover.py, distance.py, f1.py), plus integration with an external ChromaCoverId tool.
• models – model definitions and specs (Transformer‐based AMT, model_spec2midi), plus directory for parameter files (models/params/).
• utils – configuration loader, global info.
Other:
• config.json – default paths and hyperparameters.
• requirements.txt – Python dependencies.
• docs – static HTML site for project page.
README for additional insights
• Describes two workflows: “Piano Cover Generation” (inference) and “Training & Evaluation.”
• Lists dependencies, command lines, external downloads (pre-trained models, yt-dlp, dataset).
• Mentions config.json for override, and evaluation metrics from an external repo.
A. External/Raw Data Sources
1. Online audio (YouTube via yt-dlp)
2. Raw WAV files
3. MIDI files (for alignment)
B. Data Pipeline
• download.py → raw audio
• sync.py → audio/MIDI alignment
• transcribe.py → generate initial AMT outputs
• sv/extract.py → style-vector extraction
• create_labels.py → generate target labels
• create_dataset.py → build PyTorch Dataset (dataset/src.json)
C. Model Artifacts
• Pre-trained AMT model (amt.pth) in models/params
• Fine-tuned APC model (apc.pth) in models/params
D. Training Pipeline
• train/__main__.py
• _trainer.py (training loop), _loss.py
• Reads Dataset and config.json (via utils/_config.py)
E. Inference (Cover Generation)
• infer/__main__.py
• Loads APC model + pretty-midi, torchaudio, style vectors
• Outputs a new MIDI or WAV cover
F. Evaluation Pipeline
• eval/cover.py, distance.py, f1.py
• Optional integration: eval/ChromaCoverId (external repo)
G. Utilities & Configuration
• utils/_config.py, utils/_info.py, utils/info.json
• config.json at root
Relationships & Data Flows
1. External Sources → Data Pipeline → Dataset
2. Dataset + Pre-trained AMT Model → Training Pipeline → Fine-tuned APC Model
3. Fine-tuned APC Model + Input (WAV/YouTube) → Inference → Output Cover
4. Dataset or Generated Covers → Evaluation Pipeline → Metrics
Architectural patterns & design principles
• CLI-driven workflows (separate entry points for data‐prep, training, inference, evaluation)
• Modular Python packages for each concern (data, train, infer, eval, models, utils)
• Configuration via JSON + command-line overrides
• Reuse of a pre-trained AMT model (transfer learning)
• Clear separation: data ingestion vs. model logic vs. inference vs. evaluation
Detailed capture of essential elements
• Every script group (data, train, infer, eval) should appear explicitly or as sub-steps.
• Configuration store (config.json + utils) should be shown as feeding parameters into both Data and Training.
• The intermediate dataset (dataset/src.json) is a key hand-off point.
• Pre-trained vs. fine-tuned models must be distinct.