How phubbard/tgn-whisperer works
Project Type and Purpose
• This project is a command-line driven, automated pipeline that downloads podcast RSS feeds, processes audio (by calling a separate Flask-wrapped WhisperX service), parses and collates the transcription, and then publishes the results as static websites using mkdocs.
• Its main goal is to simplify podcast transcription, speaker attribution, and site generation in an incremental, robust, and maintainable way. It also handles special cases such as URL unwrapping via bit.ly resolution.
Key Components Identified from the File Structure and README
• Data Ingestion & Processing Modules:
• Files like process.py and process_podcast.py handle downloading the RSS feed (using the requests library) and parsing the XML (using xmltodict) to extract episode MP3 URLs.
• Modules (episode.py, per_podcast.py, shownotes.py) process metadata, perform speaker attribution, and collate transcription text.
• URL Resolution Component:
• The presence of unwrap-bitly.py (and the accompanying JSON dictionary file in app/bitly) indicates a specialized module that deals with resolving shortened bit.ly URLs.
• Makefile Orchestration:
• Multiple Makefiles (Makefile, create_files.mk, episode_makefile) show that the entire workflow is orchestrated via make rules that ensure robust and portable builds.
• Output & Site Generation:
• Under the “sites” directory (with subdirectories for different podcasts like “tgn” and “wcl”), mkdocs is configured to turn markdown files into static websites.
• The export file (show-notes-export.csv) and additional helper files indicate post-processing for display/publishing.
• External Transcription Service Integration:
• As noted in the README, the system calls a Flask wrapper around WhisperX (via HTTP POST) running on a separate compute server for speech transcription.
Relationships and Interactions Between Components
• The pipeline starts with downloading RSS feeds and parsing them via process.py.
• During the feed processing, if a shortened URL (bit.ly) is detected, the process interacts with the unwrap-bitly.py module to convert short URLs into canonical URLs.
• Once the MP3 URLs (or audio files) are identified, the system sends them to a dedicated transcription service (Flask-wrapped WhisperX). Data flows from the local process to this remote service via HTTP POST.
• Returned transcription data is then processed (via episode.py and related modules) that include speaker attribution and text collation.
• The processed transcription is converted into markdown formats which are fed into mkdocs (configured in the sites/tgn and sites/wcl directories) to generate a static website.
• Finally, a publish step (possibly invoked as part of the Makefile orchestration and involving rsync) updates the live static sites.
• The Makefiles serve as the central orchestrator tying these steps together reliably.
Architectural Patterns and Technologies
• Modular Pipeline: Each major step (RSS ingestion, transcription processing, URL resolution, static site generation) is encapsulated within its own module, promoting separation of concerns.
• Command-Line Driven Orchestration: The use of multiple Makefiles demonstrates an emphasis on reproducible builds and automation.
• Client-Server Integration: The Flask service for WhisperX transcription shows a client/server architecture where the local script (client) makes HTTP requests to perform heavy processing on a remote server.
• Technologies & Libraries: Python (as the main language), Requests (for HTTP), xmltodict (for RSS parsing), Flask (for the transcription API endpoint), and mkdocs (for static website generation) are all central elements.