How nkhlkp/song-identifier-v2 works
This is a Python audio recognition system packaged as a reusable library (audio_fingerprint/) with two entrypoints:
• Web app: app.py (Flask) serving templates/index.html + static/app.js|style.css, exposing REST endpoints (/add, /identify, /hum, /songs, /stats, …).
• CLI: main.py (batch operations: add/identify/list/stats/remove/clear; supports choosing a DB file via --db).
Core capability is split into two pipelines:
Hum-to-search melody extraction + embedding similarity (via Spotify basic-pitch).
Use the file tree as the authoritative map for component boundaries:
Entrypoints / Interfaces
• app.py → Flask Web Server + REST API
• main.py → CLI Application
• templates/index.html + static/app.js + static/style.css → Browser UI
Core library (the real “engine”)
Inside audio_fingerprint/:
• audio.py → audio loading & preprocessing (mono, resample, bandpass); likely invokes pydub/ffmpeg underneath
• spectrogram.py → STFT / spectrogram generation
• peaks.py → peak detection on spectrogram
• fingerprint.py → hash generation (anchor/target/time delta)
• matcher.py → lookup + time-offset histogram coherence scoring + ranking
• database.py → fingerprint store + metadata + persistence (README indicates “database file” option from CLI)
• melody.py → melody extraction (basic-pitch) + embedding + cosine similarity matching
basic-pitch inference (library/model)
Goal: show the internal modules as separate components (the repo makes this easy).
Fingerprinting pipeline components (exact match)
• audio.py AudioLoader/Preprocessor
• spectrogram.py Spectrogram/STFT
• peaks.py PeakDetector
• fingerprint.py FingerprintHasher
• database.py FingerprintDatabase (hash index + songs + stats)
• matcher.py FingerprintMatcher (offset histogram + ranker)
• pipeline.py Pipeline Orchestrator (optional but helpful: shows “glue”)
Audio file/bytes
→ Preprocess
→ STFT
→ Peaks
→ Hashes
→ (Add) store hashes into DB index
OR
→ (Identify) lookup hashes in DB index → coherence scoring → ranked results
Melody pipeline components (hum-to-search)
• melody.py MelodyExtractor (basic-pitch)
• melody.py or database.py MelodyEmbeddingStore (where embeddings live)
• melody.py SimilarityMatcher (cosine similarity)
Flow:
Hum audio
→ basic-pitch notes
→ 128-d embedding
→ compare vs stored embeddings
→ ranked matches
Web: “Add Song”
Browser (static/app.js)
→ POST /add (Flask app.py)
→ pipeline.py (or high-level API)
→ audio.py (decode via FFmpeg)
→ spectrogram.py → peaks.py → fingerprint.py
→ database.py store (hash→(song_id, t), metadata)
→ melody.py extract embedding (calls basic-pitch) [if enabled during add]
→ database.py store melody embedding
→ response (song_id, counts, status)
Web: “Hum to Search”
Browser mic recording
→ POST /hum
→ melody.py (FFmpeg decode if needed)
→ basic-pitch inference
→ embedding
→ cosine similarity vs DB embeddings
→ results
Fingerprint Hash Index
• Key: hash (anchor_freq, target_freq, dt packed)
• Value: list of (song_id, timestamp)
• This is the performance-critical index.
Song Catalog / Metadata
• song_id, title, artist, duration, fingerprint_count, etc.
Melody Embeddings Store
• song_id → 128-d normalized vector
Web UI boundary: don’t forget the frontend’s real role
From the repo:
• templates/index.html is the single-page UI shell.
• static/app.js is effectively the “frontend controller” that:
• records mic audio (browser APIs),
• uploads files (multipart/form-data),
• renders match results, library list, stats.
• That *recording happens client-side*, then audio blob is uploaded.
Diagramming conventions (to make it “principal-engineer readable”)
Use consistent notation and over-separate components (as requested), but keep it legible:
• Rectangles: services/modules
• Blue: Interfaces (Browser, CLI, REST)
• Green: Fingerprinting pipeline modules
• Purple: Melody pipeline modules
• Orange: Storage/indices
• “audio bytes (wav/mp3)”, “spectrogram (freq×time)”, “peaks (f,t)”, “hashes (uint32)”, “embedding (128f)”, “results (ranked matches)”
two dataflow lanes
• Lane A: Fingerprint add + identify
• Lane B: Melody add + hum query