# magenta/mt3 architecture

> Architecture diagram of the GitHub repository magenta/mt3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/magenta/mt3
- Repository: https://github.com/magenta/mt3

## Overview

##### Project type and purpose
- This is an open-source, research-oriented Python library & toolkit for multi-instrument music transcription (“MT3”).
- It provides data preprocessing, model definitions, training & inference interfaces (via T5X), evaluation metrics, and utility scripts, plus Colab notebooks for end-user transcription.

##### High-level file-structure analysis
- Top-level:
- .github/workflows: CI/build orchestration
- mt3/: core Python package (modules, tests, scripts, gin configs, colab notebooks)
- setup.py / setup.cfg: packaging & installation
- mt3/ subdirectories:
- colab/: interactive usage examples (Jupyter notebooks)
- gin/: experiment configurations (train, infer, eval for different datasets/models)
- scripts/: command-line helpers for dataset dumping & example extraction
- Core modules in mt3/:
- datasets.py, preprocessors.py, spectral_ops.py: data ingestion & feature extraction
- event_codec.py, run_length_encoding.py: data encoding/decoding pipelines
- network.py, models.py, layers.py: model architecture (T5 encoder/decoder stacks & custom layers)
- tasks.py: glue for T5X training tasks
- inference.py: high-level inference API
- metrics.py, metrics_utils.py, summaries.py: evaluation & logging

##### Insights from README
- Built on Google’s T5X framework for transformer-based modeling
- Two main “use cases”:
a) In-Colab inference with pretrained checkpoints (via notebooks & inference.py)
b) (Experimental) model training guided by gin configs & T5X instructions

a. Data Ingestion & Preprocessing
- Input: raw audio files
- Modules: mt3.datasets → spectral_ops → preprocessors → note_sequences
- Output: tokenized event sequences (via event_codec + run_length_encoding)
b. Model & Training Pipeline
- Gin Config Manager (mt3/gin): holds hyperparameters & file paths
- T5X Trainer (external framework)
- mt3.tasks: defines Task interface for T5X
- mt3.network / mt3.models / mt3.layers: builds Transformer network
c. Inference Pipeline
- inference.py
- Loads pretrained checkpoints
- Interacts with event_codec to decode predictions back to MIDI/note sequences
d. Evaluation & Metrics
- metrics_utils, metrics, summaries: compute transcription quality metrics & write summaries
e. CLI/Notebook Interfaces
- scripts/ (dump_task, extract_monophonic_examples)
- colab notebooks: user-facing transcription demos

##### Relationships & data flow
- Left to right (or top to bottom): Raw audio → preprocessing → dataset tokens → T5X train/infer → event decoding → MIDI/score output
- Configuration (Gin) feeds into both preprocessing and model instantiation
- Scripts & Notebooks orchestrate end-to-end pipelines by invoking the above modules

##### Key architectural patterns & design principles
- Modular ML pipeline: clear separation of data, model, training, inference, and evaluation
- Configuration-driven experiments (Gin) for reproducibility
- Adapter between T5X framework and domain-specific components (mt3.tasks, event_codec)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        A["Raw Audio (.wav)"]:::entry
        B["mt3/datasets.py"]:::core
        C["mt3/spectral_ops.py"]:::core
        D["mt3/preprocessors.py"]:::core
        E["mt3/note_sequences.py"]:::core
        F["Tokenized Event Sequences"]:::output
    end
    A --> B --> C --> D --> E --> F

    %% Event Encoding & Decoding
    subgraph "Event Encoding & Decoding"
        G["mt3/event_codec.py"]:::core
        H["mt3/run_length_encoding.py"]:::core
    end
    F --> G --> H --> I["Encoded Tokens"]:::output

    %% Configuration Management
    J["mt3/gin (Config)"]:::external

    %% Model & Training Pipeline
    subgraph "Model & Training Pipeline"
        J
        K["mt3/tasks.py"]:::core
        L["mt3/network.py"]:::core
        M["mt3/models.py"]:::core
        N["mt3/layers.py"]:::core
        O["T5X Trainer"]:::external
    end
    J --> K --> L --> M --> N --> O

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        P["mt3/inference.py"]:::core
        Q["MIDI/Note Output"]:::output
    end
    I --> P --> G --> H --> Q

    %% Evaluation & Metrics
    subgraph "Evaluation & Metrics"
        R["mt3/metrics_utils.py"]:::core
        S["mt3/metrics.py"]:::core
        T["mt3/summaries.py"]:::core
        U["Metrics Dashboard"]:::output
    end
    H --> R --> S --> T --> U

    %% Entry Points
    subgraph "Entry Points"
        V["mt3/scripts/dump_task.py"]:::entry
        W["mt3/scripts/extract_monophonic_examples.py"]:::entry
        X["mt3/colab/music_transcription_with_transformers.ipynb"]:::entry
        Y["mt3/colab/mt3_reconvat_baseline.ipynb"]:::entry
    end
    V --> B
    W --> B
    X --> P
    Y --> P

    %% Packaging & Installation
    Z["setup.py / setup.cfg"]:::external

    %% Click Events
    click B "https://github.com/magenta/mt3/blob/main/mt3/datasets.py"
    click C "https://github.com/magenta/mt3/blob/main/mt3/spectral_ops.py"
    click D "https://github.com/magenta/mt3/blob/main/mt3/preprocessors.py"
    click E "https://github.com/magenta/mt3/blob/main/mt3/note_sequences.py"
    click G "https://github.com/magenta/mt3/blob/main/mt3/event_codec.py"
    click H "https://github.com/magenta/mt3/blob/main/mt3/run_length_encoding.py"
    click J "https://github.com/magenta/mt3/tree/main/mt3/gin/"
    click K "https://github.com/magenta/mt3/blob/main/mt3/tasks.py"
    click L "https://github.com/magenta/mt3/blob/main/mt3/network.py"
    click M "https://github.com/magenta/mt3/blob/main/mt3/models.py"
    click N "https://github.com/magenta/mt3/blob/main/mt3/layers.py"
    click P "https://github.com/magenta/mt3/blob/main/mt3/inference.py"
    click R "https://github.com/magenta/mt3/blob/main/mt3/metrics_utils.py"
    click S "https://github.com/magenta/mt3/blob/main/mt3/metrics.py"
    click T "https://github.com/magenta/mt3/blob/main/mt3/summaries.py"
    click V "https://github.com/magenta/mt3/blob/main/mt3/scripts/dump_task.py"
    click W "https://github.com/magenta/mt3/blob/main/mt3/scripts/extract_monophonic_examples.py"
    click X "https://github.com/magenta/mt3/blob/main/mt3/colab/music_transcription_with_transformers.ipynb"
    click Y "https://github.com/magenta/mt3/blob/main/mt3/colab/mt3_reconvat_baseline.ipynb"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1B4F72,color:#1B2631
    classDef external fill:#E5E7E9,stroke:#566573,color:#1B2631
    classDef entry fill:#A9DFBF,stroke:#196F3D,color:#145A32
    classDef output fill:#ABEBC6,stroke:#1D8348,color:#145A32
```
