How spectraldoy/music-transformer works
This repo is a Python/PyTorch research + CLI pipeline for a Music Transformer (Transformer decoder with relative self-attention) that:
• Preprocesses single-track piano MIDI files into an event-token sequence (Oore et al.-style vocabulary)
• Trains a Transformer decoder on the resulting token tensor (with checkpointing)
• Generates new MIDI via autoregressive decoding from a trained checkpoint
• Ships pretrained model weights under models/ and a convenience notebook Generate_Music.ipynb
There is no server/runtime architecture (no backend/frontend). The “system” is best represented as offline batch pipelines + artifacts on disk.
Entry points (CLI / user-facing)
• preprocessing.py — batch job: MIDI folder → tokenized tensor .pt
• train.py — batch job: tensor .pt → checkpoints + final saved model .pt
• generate.py — inference job: saved model .pt → generated MIDI .mid
• Generate_Music.ipynb — interactive wrapper (often in Colab) around generation
Core model + NN building blocks
• model.py — Music Transformer model definition (decoder)
• layers.py — Transformer/attention/feedforward sublayers
• masking.py — causal masks / attention masking utilities
• hparams.py — hyperparameter definitions/defaults
Tokenization / vocabulary (domain layer)
• vocabulary.py — event vocabulary spec (token types / ranges)
• tokenizer.py — MIDI ↔ token IDs conversion logic
Data processing utilities
• preprocessing.py (plus any helpers inside it) — augmentation, slicing/padding, concatenation into a tensor
Artifacts / storage
• models/*.pt — pretrained state_dict + saved hparams dictionary (as described in README)
• Produced by pipeline:
• processed_data.pt (output of preprocessing)
• ckpt_path.pt (training checkpoints)
• save_path.pt (final model package: weights + hparams)
• gen_audio.mid (generated output)
External dependencies (critical “external systems”)
• PyTorch (torch) — training/inference runtime, tensor serialization
• Mido — MIDI I/O parsing/writing
• (Optional) GPU/CUDA environment — compute substrate
• Filesystem — the main integration surface (inputs/outputs are files)
Preprocessing Job (preprocessing.py)
• Inputs: .../datapath/ (MIDI files), params (lth, augmentation settings)
• Uses: tokenizer.py, vocabulary.py, Mido
• Outputs: processed_data.pt (single concatenated PyTorch tensor)
Training Job (train.py + MusicTransformerTrainer)
• Inputs: processed_data.pt, hparams (from hparams.py or loaded from checkpoint)
• Uses: model.py, layers.py, masking.py, PyTorch
• Outputs:
• ckpt_path.pt (optimizer + scheduler + weights + hparams)
• save_path.pt (final package: state_dict + hparams dict)
Generation Job (generate.py)
• Inputs: save_path.pt (or one of models/*.pt)
• Uses: model.py to instantiate model from hparams, tokenizer to detokenize, Mido to write MIDI
• Outputs: gen_audio.mid
• Parameters: decoding strategy (argmax/categorical), temperature, top_k, tempo
Interactive Notebook (Generate_Music.ipynb)
• Calls generation logic (either shelling out to generate.py or importing functions)
• (Optionally) mentions using Magenta SoundFont for playback in Colab (even if not in file tree, it’s a runtime integration mentioned in README)
Goal: show internal modules and how they connect, without drowning in math.
C1) Tokenization/Domain pipeline (MIDI ↔ tokens)
• vocabulary.py defines token space (events)
• tokenizer.py:
• MIDI → event sequence → token IDs
• token IDs → event sequence → MIDI
• preprocessing.py:
• window sampling / padding to max length
• augmentation (pitch transpose, time stretch)
• concatenation into a training tensor
MIDI → Tokenizer → token IDs → (augment/slice/pad) → dataset tensor
C2) Model internals (Transformer decoder)
• model.py (MusicTransformer)
• layers.py (decoder blocks: attention + FFN + norms)
• masking.py (causal attention mask)
• (Optionally) hparams.py feeding configuration into model construction
token IDs → embedding → N decoder blocks (relative attention) → logits over vocabulary → loss during training / sampling during inference
• MIDI files (.mid) → Preprocessing
• Preprocessing → processed_data.pt (Torch tensor)
• processed_data.pt → Training
• Training → ckpt_path.pt (checkpoint: weights+optim+scheduler+hparams)
• Training → save_path.pt (model package: state_dict+hparams)
• save_path.pt → Generation
• Generation → gen_audio.mid
• Preprocessing uses tokenizer.py, vocabulary.py, mido
• Training uses model.py, layers.py, masking.py, torch
• Generation uses model.py, tokenizer.py, mido
Architectural patterns/design principles visible here (label them)
• Artifact-based pipeline: modules communicate via .pt and .mid files, not via RPC.
• Separation of concerns:
• Token domain logic (vocabulary.py/tokenizer.py) decoupled from model (model.py)
• Training orchestration (train.py) decoupled from inference (generate.py)
• Checkpoint/resume training: trainer persists full state (model + optimizer + scheduler + hparams)
• Autoregressive decoding: generation is sequential; sampling parameters are part of runtime config.
• PyTorch 2.1.0: training loop, model, serialization (.pt)
• Mido 1.2.9: MIDI read/write