# toxtli/hum2song architecture

> Architecture diagram of the GitHub repository toxtli/hum2song, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/toxtli/hum2song
- Repository: https://github.com/toxtli/hum2song

## Overview

##### Project Type and Purpose
- This is a client-side–only, static single-page web application (no server/backend).
- Purpose: record a user’s voice, transcribe the melody with a neural network, generate multi-track accompaniment, and play it back—all in the browser.

##### File Structure
- index.html / index.js / index.css: entry point and UI logic.
- assets/libs: third-party JS libraries (Magenta.js, Tone.js, WebMIDI, JQuery, Lodash, MidiConvert, FileSaver, etc.).
- assets/models: pre-trained TensorFlow.js / Magenta models (basic_rnn, chord_pitches_improv, onsets_frames_uni, etc.).
- assets/fonts & assets/images: static styling assets.

##### README Insights
- All processing happens in-browser.
- Models (~100 MB) load over HTTP when the SPA boots.
- Uses Web Audio API + Tone.js for recording/playback, and Magenta / TF.js for ML.

##### Main Components & Interactions

a. User Interface (UI) Layer
– index.html & CSS: record/stop buttons, progress indicators.
– index.js: orchestrates user events and ML pipelines.

b. Audio Input Module
– Web Audio API (in browser): captures microphone stream.
– Passes raw audio buffers to the Transcription Engine.

c. Melody Transcription Engine
– assets/models/onsets_frames_uni via Magenta.js / tfjs.
– Converts audio → note/onset events.

d. Music Generation Engines
– assets/models/basic_rnn, chord_pitches_improv, multitrack_chords, trio_4bar, genre models.
– Receives note events and generates multi-track MIDI sequences.

e. MIDI Processing Layer
– MidiConvert.js: serialize/deserialize MIDI data.
– FileSaver.js: export generated MIDI files.

f. Synthesis & Playback Engine
– Tone.js / WebMIDI: schedule and play back generated tracks.
– assets/libs/synthjs or midijs for sound fonts.

g. Static Asset Server
– All assets/models served over HTTP (Python SimpleHTTPServer or any static host).

##### Architectural Patterns and Principles
- Pipeline pattern: Audio→Transcription→Generation→Synthesis.
- Separation of concerns: UI vs. ML vs. Playback vs. Serialization.
- Client-heavy ML: models loaded at runtime in-browser.

##### Technologies & Frameworks to Note
- Magenta.js & TensorFlow.js – ML inference.
- Web Audio API & Tone.js – recording & playback.
- MidiConvert.js & FileSaver.js – MIDI I/O.
- jQuery & Lodash – DOM utilities.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Browser Boundary
    subgraph "Browser (Client-side)"
        subgraph "UI Layer"
            UI_HTML["index.html"]:::frontend
            UI_CSS["index.css"]:::frontend
            UI_JS["index.js"]:::frontend
        end
        subgraph "Audio Input Module"
            Audio["Web Audio API\n(captured in index.js)"]:::audio
        end
        subgraph "Melody Transcription Engine"
            TransModel["onsets_frames_uni\n(model files)"]:::ml
        end
        subgraph "Music Generation Engines"
            BasicRNN["basic_rnn"]:::ml
            ChordImp["chord_pitches_improv"]:::ml
            MultiChord["multitrack_chords"]:::ml
            Trio["trio_4bar"]:::ml
            Genre["genre"]:::ml
        end
        subgraph "MIDI Processing Layer"
            MidiConvert["MidiConvert.js"]:::midi
            FileSaver["FileSaver.js"]:::midi
        end
        subgraph "Synthesis & Playback Engine"
            Tone["Tone.js"]:::playback
            SynthJS["synth.js"]:::playback
            MidiJS["midijs"]:::playback
            WebMIDI["WebMIDI API"]:::playback
        end
        subgraph "Third-Party Utilities"
            jQuery["jQuery"]:::util
            Lodash["Lodash"]:::util
        end
    end

    subgraph "Static Assets Server"
        AssetsFonts["assets/fonts"]:::assets
        AssetsImages["assets/images"]:::assets
        AssetsModels["assets/models"]:::assets
        AssetsLibs["assets/libs"]:::assets
    end

    %% Data Flow
    User["User"]:::external
    User -->|interacts| UI_JS
    UI_JS -->|"AudioBuffer"| Audio
    Audio -->|"AudioBuffer"| TransModel
    TransModel -->|"Note[]"| BasicRNN
    TransModel -->|"Note[]"| ChordImp
    TransModel -->|"Note[]"| MultiChord
    TransModel -->|"Note[]"| Trio
    TransModel -->|"Note[]"| Genre
    BasicRNN -->|"MIDI Blob"| MidiConvert
    ChordImp -->|"MIDI Blob"| MidiConvert
    MultiChord -->|"MIDI Blob"| MidiConvert
    Trio -->|"MIDI Blob"| MidiConvert
    Genre -->|"MIDI Blob"| MidiConvert
    MidiConvert -->|"MIDI Data"| Tone
    MidiConvert -->|"MIDI Blob"| FileSaver
    Tone -->|"Audio Playback"| SynthJS
    Tone -->|"MIDI Playback"| MidiJS
    Tone -->|"MIDI Playback"| WebMIDI
    FileSaver -->|download| User

    %% Asset Loading
    AssetsModels --> TransModel
    AssetsModels --> BasicRNN
    AssetsModels --> ChordImp
    AssetsModels --> MultiChord
    AssetsModels --> Trio
    AssetsModels --> Genre
    AssetsLibs --> TransModel
    AssetsLibs --> BasicRNN
    AssetsLibs --> MidiConvert
    AssetsLibs --> FileSaver
    AssetsLibs --> Tone
    AssetsLibs --> SynthJS
    AssetsLibs --> MidiJS
    AssetsLibs --> WebMIDI
    AssetsLibs --> jQuery
    AssetsLibs --> Lodash

    %% Click Events
    click UI_HTML "https://github.com/toxtli/hum2song/blob/master/index.html"
    click UI_CSS "https://github.com/toxtli/hum2song/blob/master/index.css"
    click UI_JS "https://github.com/toxtli/hum2song/blob/master/index.js"
    click TransModel "https://github.com/toxtli/hum2song/tree/master/assets/models/onsets_frames_uni/"
    click BasicRNN "https://github.com/toxtli/hum2song/tree/master/assets/models/basic_rnn/"
    click ChordImp "https://github.com/toxtli/hum2song/tree/master/assets/models/chord_pitches_improv/"
    click MultiChord "https://github.com/toxtli/hum2song/tree/master/assets/models/multitrack_chords/"
    click Trio "https://github.com/toxtli/hum2song/tree/master/assets/models/trio_4bar/"
    click Genre "https://github.com/toxtli/hum2song/tree/master/assets/models/genre/"
    click MidiConvert "https://github.com/toxtli/hum2song/blob/master/assets/libs/midiconvert@0.4.7/MidiConvert.js"
    click FileSaver "https://github.com/toxtli/hum2song/tree/master/assets/libs/FileSaver@1.3.8/"
    click Tone "https://github.com/toxtli/hum2song/blob/master/assets/libs/tonejs/Tone.js"
    click SynthJS "https://github.com/toxtli/hum2song/blob/master/assets/libs/synthjs/synth.js"
    click MidiJS "https://github.com/toxtli/hum2song/tree/master/assets/libs/midijs/"
    click WebMIDI "https://github.com/toxtli/hum2song/tree/master/assets/libs/webmidi@2.0.0/"
    click jQuery "https://github.com/toxtli/hum2song/blob/master/assets/libs/jquery@3.3.1/jquery.js"
    click Lodash "https://github.com/toxtli/hum2song/tree/master/assets/libs/lodash@4.17.4/"
    click AssetsFonts "https://github.com/toxtli/hum2song/tree/master/assets/fonts/"
    click AssetsImages "https://github.com/toxtli/hum2song/tree/master/assets/images/"
    click AssetsModels "https://github.com/toxtli/hum2song/tree/master/assets/models/"
    click AssetsLibs "https://github.com/toxtli/hum2song/tree/master/assets/libs/"

    %% Styles
    classDef frontend fill:#E0F7FA,stroke:#006064,color:#000;
    classDef audio fill:#E8F5E9,stroke:#1B5E20,color:#000;
    classDef ml fill:#F3E5F5,stroke:#4A148C,color:#000;
    classDef midi fill:#FFF3E0,stroke:#E65100,color:#000;
    classDef playback fill:#E1F5FE,stroke:#01579B,color:#000;
    classDef util fill:#ECEFF1,stroke:#455A64,color:#000;
    classDef assets fill:#EFEBE9,stroke:#3E2723,color:#000;
    classDef external fill:#FFFFFF,stroke:#BDBDBD,color:#000;
```
