How toxtli/hum2song works
Project Type and Purpose
• This is a client-side–only, static single-page web application (no server/backend).
• Purpose: record a user’s voice, transcribe the melody with a neural network, generate multi-track accompaniment, and play it back—all in the browser.
File Structure
• index.html / index.js / index.css: entry point and UI logic.
• assets/libs: third-party JS libraries (Magenta.js, Tone.js, WebMIDI, JQuery, Lodash, MidiConvert, FileSaver, etc.).
• assets/models: pre-trained TensorFlow.js / Magenta models (basic_rnn, chord_pitches_improv, onsets_frames_uni, etc.).
• assets/fonts & assets/images: static styling assets.
README Insights
• All processing happens in-browser.
• Models (~100 MB) load over HTTP when the SPA boots.
• Uses Web Audio API + Tone.js for recording/playback, and Magenta / TF.js for ML.
Main Components & Interactions
a. User Interface (UI) Layer
– index.html & CSS: record/stop buttons, progress indicators.
– index.js: orchestrates user events and ML pipelines.
b. Audio Input Module
– Web Audio API (in browser): captures microphone stream.
– Passes raw audio buffers to the Transcription Engine.
c. Melody Transcription Engine
– assets/models/onsets_frames_uni via Magenta.js / tfjs.
– Converts audio → note/onset events.
d. Music Generation Engines
– assets/models/basic_rnn, chord_pitches_improv, multitrack_chords, trio_4bar, genre models.
– Receives note events and generates multi-track MIDI sequences.
e. MIDI Processing Layer
– MidiConvert.js: serialize/deserialize MIDI data.
– FileSaver.js: export generated MIDI files.
f. Synthesis & Playback Engine
– Tone.js / WebMIDI: schedule and play back generated tracks.
– assets/libs/synthjs or midijs for sound fonts.
g. Static Asset Server
– All assets/models served over HTTP (Python SimpleHTTPServer or any static host).
Architectural Patterns and Principles
• Pipeline pattern: Audio→Transcription→Generation→Synthesis.
• Separation of concerns: UI vs. ML vs. Playback vs. Serialization.
• Client-heavy ML: models loaded at runtime in-browser.
Technologies & Frameworks to Note
• Magenta.js & TensorFlow.js – ML inference.
• Web Audio API & Tone.js – recording & playback.
• MidiConvert.js & FileSaver.js – MIDI I/O.
• jQuery & Lodash – DOM utilities.