How devsp1312/serendipity works
Serendipity is a single-machine, fully local, two-stage ML pipeline that:
• Ingests audio files from a local folder
• Runs Stage 1 transcription (Faster-Whisper) to produce timestamped text
• Runs Stage 2 analysis (multiple NLP/audio models) to generate:
• audio emotion, text emotion (27 labels), likes/dislikes, “person type”
• HTML report
• optional longitudinal speaker profiles (19 metrics across sessions)
• Exposes a local Flask Web UI that orchestrates the pipeline and streams progress via SSE (Server-Sent Events)
So the “system” is not a distributed architecture; it’s a local app composed of CLI + local web server + ML inference modules + on-disk artifacts + cached models.
• The two-stage pipeline boundary
• The file-based handoffs/caching
• The orchestration paths (Web UI vs CLI)
• The model caches and “download models” bootstrapping steps
• The outputs (JSON, reports, profiles) and how they feed back into later sessions
Derive the architecture from the file tree + README (what to include)
From the README structure:
Core runtime components
• Stage 1 (transcription): stage1/transcribe.py, stage1/download_model.py, stage1/model/ cache
• Stage 2 (analysis + UI):
• Web server: stage2/app.py (Flask + SSE streaming)
• CLI analysis: stage2/analyse.py
• Analysis modules:
• audio_emotion.py (wav2vec2 emotion)
• text_emotion.py (GoEmotions BERT)
• likes_dislikes.py (sentiment + topic extraction)
• person_type.py (zero-shot NLI)
• profiles.py (19-metric aggregation across sessions)
• Model bootstrap: download_models.py, stage2/models/ cache
Data / artifact directories (important: these are “storage subsystems”)
• Audio files/ (input)
• Text files/ (transcripts + segments)
• JSON files/ (analysis outputs)
• Reports/ (HTML output)
• Profiles/ (long-lived aggregated speaker profiles)
External dependencies (represent as “libraries/models,” not network services)
• HuggingFace/transformers model artifacts (downloaded once, then offline)
• Faster-Whisper model (downloaded once, then offline)
• Hardware acceleration optional: CPU default; GPU if available (MPS/CUDA)
Optional: add to profile → update Profiles/<speaker>.json
SSE progress stream from Flask back to browser during steps.
Architectural patterns worth explicitly highlighting (because they’re key here)
• Two-stage pipeline with a hard boundary:
• Stage 1: speech-to-text
• Stage 2: multi-model analysis + reporting + profiles
• File-based caching and idempotency:
• transcript cache means re-analysis skips Stage 1
• Local-first / offline-first:
• downloads occur once; runtime is offline
• Orchestration via UI:
• Flask app orchestrates and provides a progress/event channel (SSE)
• Pluggable step modules:
User & Interfaces
• User
• Browser (index.html)
• CLI terminal (optional branch)
Orchestrator
• Flask Web Server stage2/app.py
Pipeline (boxed as ‘Analysis Pipeline’)
• Step 1: Transcription stage1/transcribe.py (with “skip if cached” note)
• Step 2: audio_emotion.py
• Step 3: text_emotion.py
• Step 4: likes_dislikes.py
• Step 5: person_type.py
• Step 6: profiles.py (conditional on “Add to Profile” or profile mode)
• Audio files/
• Text files/
• JSON files/
• Reports/
• Profiles/
Model caches (side row)
• stage1/model (Whisper)
• stage2/models (wav2vec2, goemotions, sentiment, zero-shot)
This layout matches how the repo is mentally operated: UI/CLI triggers → pipeline steps → filesystem artifacts → UI displays.
Don’t overthink it: maximize components (as requested)
• Keep each folder as its own storage node
• Distinguish “model download” scripts from “runtime inference” modules
• SSE as an explicit channel, not an implementation detail