# spectraldoy/music-transformer architecture

> Architecture diagram of the GitHub repository spectraldoy/music-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spectraldoy/music-transformer
- Repository: https://github.com/spectraldoy/music-transformer

## Overview

This repo is a **Python/PyTorch research + CLI pipeline** for a **Music Transformer (Transformer decoder with relative self-attention)** that:
- **Preprocesses** single-track piano **MIDI files** into an **event-token sequence** (Oore et al.-style vocabulary)
- **Trains** a Transformer decoder on the resulting token tensor (with checkpointing)
- **Generates** new MIDI via autoregressive decoding from a trained checkpoint
- Ships **pretrained model weights** under `models/` and a convenience notebook `Generate_Music.ipynb`

There is **no server/runtime architecture** (no backend/frontend). The “system” is best represented as **offline batch pipelines + artifacts on disk**.

##### Entry points (CLI / user-facing)
- `preprocessing.py` — batch job: MIDI folder → tokenized tensor `.pt`
- `train.py` — batch job: tensor `.pt` → checkpoints + final saved model `.pt`
- `generate.py` — inference job: saved model `.pt` → generated MIDI `.mid`
- `Generate_Music.ipynb` — interactive wrapper (often in Colab) around generation

##### Core model + NN building blocks
- `model.py` — Music Transformer model definition (decoder)
- `layers.py` — Transformer/attention/feedforward sublayers
- `masking.py` — causal masks / attention masking utilities
- `hparams.py` — hyperparameter definitions/defaults

##### Tokenization / vocabulary (domain layer)
- `vocabulary.py` — event vocabulary spec (token types / ranges)
- `tokenizer.py` — MIDI ↔ token IDs conversion logic

##### Data processing utilities
- `preprocessing.py` (plus any helpers inside it) — augmentation, slicing/padding, concatenation into a tensor

##### Artifacts / storage
- `models/*.pt` — pretrained `state_dict` + saved `hparams` dictionary (as described in README)
- Produced by pipeline:
- `processed_data.pt` (output of preprocessing)
- `ckpt_path.pt` (training checkpoints)
- `save_path.pt` (final model package: weights + hparams)
- `gen_audio.mid` (generated output)

##### External dependencies (critical “external systems”)
- PyTorch (`torch`) — training/inference runtime, tensor serialization
- Mido — MIDI I/O parsing/writing
- (Optional) GPU/CUDA environment — compute substrate
- Filesystem — the main integration surface (inputs/outputs are files)

##### Preprocessing Job (`preprocessing.py`)
- Inputs: `.../datapath/` (MIDI files), params (`lth`, augmentation settings)
- Uses: `tokenizer.py`, `vocabulary.py`, Mido
- Outputs: `processed_data.pt` (single concatenated PyTorch tensor)

##### Training Job (`train.py` + `MusicTransformerTrainer`)
- Inputs: `processed_data.pt`, `hparams` (from `hparams.py` or loaded from checkpoint)
- Uses: `model.py`, `layers.py`, `masking.py`, PyTorch
- Outputs:
- `ckpt_path.pt` (optimizer + scheduler + weights + hparams)
- `save_path.pt` (final package: `state_dict` + `hparams` dict)

##### Generation Job (`generate.py`)
- Inputs: `save_path.pt` (or one of `models/*.pt`)
- Uses: `model.py` to instantiate model from hparams, tokenizer to detokenize, Mido to write MIDI
- Outputs: `gen_audio.mid`
- Parameters: decoding strategy (`argmax`/`categorical`), temperature, top_k, tempo

##### Interactive Notebook (`Generate_Music.ipynb`)
- Calls generation logic (either shelling out to `generate.py` or importing functions)
- (Optionally) mentions using Magenta SoundFont for playback in Colab (even if not in file tree, it’s a runtime integration mentioned in README)

Goal: show internal modules and how they connect, without drowning in math.

##### C1) Tokenization/Domain pipeline (MIDI ↔ tokens)
- `vocabulary.py` defines token space (events)
- `tokenizer.py`:
- MIDI → event sequence → token IDs
- token IDs → event sequence → MIDI
- `preprocessing.py`:
- window sampling / padding to max length
- augmentation (pitch transpose, time stretch)
- concatenation into a training tensor

MIDI → Tokenizer → token IDs → (augment/slice/pad) → dataset tensor

##### C2) Model internals (Transformer decoder)
- `model.py` (MusicTransformer)
- `layers.py` (decoder blocks: attention + FFN + norms)
- `masking.py` (causal attention mask)
- (Optionally) `hparams.py` feeding configuration into model construction

token IDs → embedding → N decoder blocks (relative attention) → logits over vocabulary → loss during training / sampling during inference

- `MIDI files (.mid)` → **Preprocessing**
- Preprocessing → `processed_data.pt` (**Torch tensor**)
- `processed_data.pt` → **Training**
- Training → `ckpt_path.pt` (**checkpoint: weights+optim+scheduler+hparams**)
- Training → `save_path.pt` (**model package: state_dict+hparams**)
- `save_path.pt` → **Generation**
- Generation → `gen_audio.mid`

- Preprocessing uses `tokenizer.py`, `vocabulary.py`, `mido`
- Training uses `model.py`, `layers.py`, `masking.py`, `torch`
- Generation uses `model.py`, `tokenizer.py`, `mido`

##### Architectural patterns/design principles visible here (label them)

- **Artifact-based pipeline**: modules communicate via `.pt` and `.mid` files, not via RPC.
- **Separation of concerns**:
- Token domain logic (`vocabulary.py`/`tokenizer.py`) decoupled from model (`model.py`)
- Training orchestration (`train.py`) decoupled from inference (`generate.py`)
- **Checkpoint/resume training**: trainer persists full state (model + optimizer + scheduler + hparams)
- **Autoregressive decoding**: generation is sequential; sampling parameters are part of runtime config.

- **PyTorch 2.1.0**: training loop, model, serialization (`.pt`)
- **Mido 1.2.9**: MIDI read/write

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ========== LEVEL 0: SYSTEM CONTEXT ==========
    subgraph "Level 0 — System Context (Offline, Artifact-Based)"
        U["User / Researcher"]:::actor

        ENV["CPU/GPU Runtime\n(Python + CUDA optional)"]:::external
        FS[("Local Filesystem / Artifact Store")]:::storage

        MIDIIN["MIDI Dataset Folder\n(.mid files)"]:::external
        COLAB["(Optional) Google Colab Runtime\nfor notebook use"]:::external
    end

    %% ========== LEVEL 1: MAIN PIPELINE ==========
    subgraph "Level 1 — Batch Pipeline (CLI + Notebook)"
        direction TB

        PRE["Preprocessing Job\n(preprocessing.py)"]:::job
        A_PROC["processed_data.pt\n(token tensor)"]:::artifact

        TRN["Training Job\n(train.py + MusicTransformerTrainer)"]:::job
        A_CKPT["ckpt_path.pt\n(checkpoint: weights+optim+scheduler+hparams)"]:::artifact
        A_SAVE["save_path.pt\n(model package: state_dict+hparams)"]:::artifact

        GEN["Generation Job\n(generate.py)"]:::job
        A_MIDIOUT["gen_audio.mid\n(generated MIDI)"]:::artifact

        NB["Interactive Notebook\n(Generate_Music.ipynb)"]:::notebook

        MODELDIR["Pretrained Models Directory\n(models/*.pt)"]:::storage
        M1["chopintransformerv5.pt"]:::artifact
        M2["model4v2.pt"]:::artifact
        M3["model6v2.pt"]:::artifact
        M4["vgmtransformerv4.pt"]:::artifact
    end

    %% ========== LEVEL 2: COMPONENTS ==========
    subgraph "Level 2 — Components (Domain + Model)"
        direction TB

        subgraph "Tokenization / Domain Layer (MIDI ↔ Tokens)"
            direction TB
            VOC["Event Vocabulary Spec\n(vocabulary.py)"]:::module
            TOK["Tokenizer\n(MIDI ↔ token IDs)\n(tokenizer.py)"]:::module
            DEP_MIDO["Mido 1.2.9\n(MIDI read/write)"]:::external
        end

        subgraph "Model Layer (Music Transformer Decoder)"
            direction TB
            HP["Hyperparameters / Defaults\n(hparams.py)"]:::module
            MOD["Music Transformer\n(Transformer decoder + relative self-attention)\n(model.py)"]:::module
            LAY["Transformer Sublayers\n(attention/FFN/norms)\n(layers.py)"]:::module
            MSK["Causal / Attention Masking\n(masking.py)"]:::module
            DEP_TORCH["PyTorch 2.1.0\n(model + training + serialization)"]:::external
        end

        REQ["Project Dependencies\n(requirements.txt)"]:::module
    end

    %% ========== DATA FLOW (SOLID) ==========
    U -->|"runs"| PRE
    U -->|"runs"| TRN
    U -->|"runs"| GEN
    U -->|"uses"| NB

    MIDIIN -->|"MIDI files (.mid)"| PRE
    PRE -->|"token tensor (.pt)"| A_PROC
    A_PROC -->|"training data (.pt)"| TRN
    TRN -->|"checkpoint (.pt)"| A_CKPT
    TRN -->|"final model package (.pt)"| A_SAVE
    A_SAVE -->|"model package (.pt)"| GEN
    GEN -->|"generated MIDI (.mid)"| A_MIDIOUT

    %% Alternate model source: pretrained models directory
    MODELDIR -->|"select pretrained (.pt)"| GEN
    MODELDIR --> M1
    MODELDIR --> M2
    MODELDIR --> M3
    MODELDIR --> M4
    M1 -->|"model package (.pt)"| GEN
    M2 -->|"model package (.pt)"| GEN
    M3 -->|"model package (.pt)"| GEN
    M4 -->|"model package (.pt)"| GEN

    %% Notebook orchestration paths
    COLAB -.->|"runtime option"| NB
    NB -->|"invokes generation"| GEN

    %% Filesystem as integration surface
    PRE -->|"writes"| FS
    TRN -->|"writes"| FS
    GEN -->|"writes"| FS
    FS -->|"reads"| TRN
    FS -->|"reads"| GEN

    %% ========== CODE DEPENDENCIES (DASHED) ==========
    PRE -.->|"uses"| TOK
    PRE -.->|"uses"| VOC
    PRE -.->|"uses"| DEP_MIDO
    PRE -.->|"executes on"| ENV
    PRE -.->|"reads/writes via"| DEP_TORCH

    TRN -.->|"uses"| HP
    TRN -.->|"uses"| MOD
    TRN -.->|"uses"| LAY
    TRN -.->|"uses"| MSK
    TRN -.->|"uses"| DEP_TORCH
    TRN -.->|"executes on"| ENV

    GEN -.->|"uses"| MOD
    GEN -.->|"uses"| HP
    GEN -.->|"uses"| TOK
    GEN -.->|"uses"| VOC
    GEN -.->|"uses"| DEP_MIDO
    GEN -.->|"uses"| DEP_TORCH
    GEN -.->|"executes on"| ENV

    TOK -.->|"references token space"| VOC
    MOD -.->|"built from"| LAY
    MOD -.->|"applies"| MSK
    MOD -.->|"configured by"| HP
    MOD -.->|"implemented with"| DEP_TORCH

    REQ -.->|"pins versions for"| DEP_TORCH
    REQ -.->|"pins versions for"| DEP_MIDO

    %% ========== ANNOTATIONS ==========
    N1["Design Notes:\n• Artifact-based pipeline (.mid/.pt)\n• Separation: token domain vs model\n• Checkpoint/resume: full trainer state\n• Autoregressive decoding with runtime sampling params"]:::note
    FS --- N1

    %% ========== CLICK EVENTS (from component_mapping) ==========
    click PRE "https://github.com/spectraldoy/music-transformer/blob/main/preprocessing.py"
    click TRN "https://github.com/spectraldoy/music-transformer/blob/main/train.py"
    click GEN "https://github.com/spectraldoy/music-transformer/blob/main/generate.py"
    click NB "https://github.com/spectraldoy/music-transformer/blob/main/Generate_Music.ipynb"

    click MOD "https://github.com/spectraldoy/music-transformer/blob/main/model.py"
    click LAY "https://github.com/spectraldoy/music-transformer/blob/main/layers.py"
    click MSK "https://github.com/spectraldoy/music-transformer/blob/main/masking.py"
    click HP "https://github.com/spectraldoy/music-transformer/blob/main/hparams.py"

    click VOC "https://github.com/spectraldoy/music-transformer/blob/main/vocabulary.py"
    click TOK "https://github.com/spectraldoy/music-transformer/blob/main/tokenizer.py"

    click MODELDIR "https://github.com/spectraldoy/music-transformer/tree/main/models/"
    click M1 "https://github.com/spectraldoy/music-transformer/blob/main/models/chopintransformerv5.pt"
    click M2 "https://github.com/spectraldoy/music-transformer/blob/main/models/model4v2.pt"
    click M3 "https://github.com/spectraldoy/music-transformer/blob/main/models/model6v2.pt"
    click M4 "https://github.com/spectraldoy/music-transformer/blob/main/models/vgmtransformerv4.pt"

    click REQ "https://github.com/spectraldoy/music-transformer/blob/main/requirements.txt"

    %% ========== STYLES ==========
    classDef actor fill:#111827,stroke:#0f172a,color:#ffffff,stroke-width:2px
    classDef job fill:#2563eb,stroke:#1e40af,color:#ffffff,stroke-width:2px
    classDef notebook fill:#7c3aed,stroke:#5b21b6,color:#ffffff,stroke-width:2px
    classDef module fill:#10b981,stroke:#047857,color:#06281f,stroke-width:2px
    classDef artifact fill:#f59e0b,stroke:#b45309,color:#1f1300,stroke-width:2px
    classDef storage fill:#14b8a6,stroke:#0f766e,color:#062b2a,stroke-width:2px
    classDef external fill:#e5e7eb,stroke:#6b7280,color:#111827,stroke-width:1.5px
    classDef note fill:#fff7ed,stroke:#c2410c,color:#7c2d12,stroke-width:1.5px,stroke-dasharray: 4 3
```
