# phubbard/tgn-whisperer architecture

> Architecture diagram of the GitHub repository phubbard/tgn-whisperer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phubbard/tgn-whisperer
- Repository: https://github.com/phubbard/tgn-whisperer

## Overview

##### Project Type and Purpose
- This project is a command-line driven, automated pipeline that downloads podcast RSS feeds, processes audio (by calling a separate Flask-wrapped WhisperX service), parses and collates the transcription, and then publishes the results as static websites using mkdocs.
- Its main goal is to simplify podcast transcription, speaker attribution, and site generation in an incremental, robust, and maintainable way. It also handles special cases such as URL unwrapping via bit.ly resolution.

##### Key Components Identified from the File Structure and README
- Data Ingestion & Processing Modules:
- Files like process.py and process_podcast.py handle downloading the RSS feed (using the requests library) and parsing the XML (using xmltodict) to extract episode MP3 URLs.
- Modules (episode.py, per_podcast.py, shownotes.py) process metadata, perform speaker attribution, and collate transcription text.
- URL Resolution Component:
- The presence of unwrap-bitly.py (and the accompanying JSON dictionary file in app/bitly) indicates a specialized module that deals with resolving shortened bit.ly URLs.
- Makefile Orchestration:
- Multiple Makefiles (Makefile, create_files.mk, episode_makefile) show that the entire workflow is orchestrated via make rules that ensure robust and portable builds.
- Output & Site Generation:
- Under the “sites” directory (with subdirectories for different podcasts like “tgn” and “wcl”), mkdocs is configured to turn markdown files into static websites.
- The export file (show-notes-export.csv) and additional helper files indicate post-processing for display/publishing.
- External Transcription Service Integration:
- As noted in the README, the system calls a Flask wrapper around WhisperX (via HTTP POST) running on a separate compute server for speech transcription.

##### Relationships and Interactions Between Components
- The pipeline starts with downloading RSS feeds and parsing them via process.py.
- During the feed processing, if a shortened URL (bit.ly) is detected, the process interacts with the unwrap-bitly.py module to convert short URLs into canonical URLs.
- Once the MP3 URLs (or audio files) are identified, the system sends them to a dedicated transcription service (Flask-wrapped WhisperX). Data flows from the local process to this remote service via HTTP POST.
- Returned transcription data is then processed (via episode.py and related modules) that include speaker attribution and text collation.
- The processed transcription is converted into markdown formats which are fed into mkdocs (configured in the sites/tgn and sites/wcl directories) to generate a static website.
- Finally, a publish step (possibly invoked as part of the Makefile orchestration and involving rsync) updates the live static sites.
- The Makefiles serve as the central orchestrator tying these steps together reliably.

##### Architectural Patterns and Technologies
- Modular Pipeline: Each major step (RSS ingestion, transcription processing, URL resolution, static site generation) is encapsulated within its own module, promoting separation of concerns.
- Command-Line Driven Orchestration: The use of multiple Makefiles demonstrates an emphasis on reproducible builds and automation.
- Client-Server Integration: The Flask service for WhisperX transcription shows a client/server architecture where the local script (client) makes HTTP requests to perform heavy processing on a remote server.
- Technologies & Libraries: Python (as the main language), Requests (for HTTP), xmltodict (for RSS parsing), Flask (for the transcription API endpoint), and mkdocs (for static website generation) are all central elements.

## Diagram (Mermaid)

```mermaid
flowchart TD
    MA["Build Automation (Makefiles)"]:::orchestration
    RSS["RSS Feed Downloader & Parser"]:::ingestion
    BITLY["Bit.ly URL Resolver"]:::ingestion
    TPROC["Transcription Processor"]:::processing
    FAPI["Flask WhisperX Service"]:::external
    MD["Markdown Converter"]:::processing
    SITE["Static Site Generator (mkdocs)"]:::siteGen
    PUB["Publish/Deployment (rsync)"]:::siteGen
    ARCH["Archive & Logging"]:::ingestion

    %% Orchestration Connections
    MA ---|"triggers"| RSS
    MA ---|"triggers"| BITLY
    MA ---|"triggers"| TPROC
    MA ---|"triggers"| SITE

    %% Main Pipeline Flow
    RSS -->|"ExtractMP3URLs"| TPROC
    RSS -->|"ArchiveRSS"| ARCH
    RSS -->|"DetectShortURL"| BITLY
    BITLY -->|"ResolvedURL"| TPROC
    TPROC -->|"HTTPPOST"| FAPI
    TPROC -->|"CollateTranscript"| MD
    MD -->|"GenerateMarkdown"| SITE
    SITE -->|"DeploySite"| PUB

    %% Click Events for Component Mapping
    click RSS "https://github.com/phubbard/tgn-whisperer/blob/main/app/process.py"
    click RSS "https://github.com/phubbard/tgn-whisperer/blob/main/process_podcast.py"
    click BITLY "https://github.com/phubbard/tgn-whisperer/blob/main/app/unwrap-bitly.py"
    click BITLY "https://github.com/phubbard/tgn-whisperer/blob/main/app/bitly/bitly.json"
    click TPROC "https://github.com/phubbard/tgn-whisperer/blob/main/app/episode.py"
    click TPROC "https://github.com/phubbard/tgn-whisperer/blob/main/app/per_podcast.py"
    click TPROC "https://github.com/phubbard/tgn-whisperer/blob/main/app/shownotes.py"
    click SITE "https://github.com/phubbard/tgn-whisperer/tree/main/sites/tgn"
    click SITE "https://github.com/phubbard/tgn-whisperer/tree/main/sites/wcl"
    click MA "https://github.com/phubbard/tgn-whisperer/tree/main/Makefile"
    click MA "https://github.com/phubbard/tgn-whisperer/blob/main/create_files.mk"
    click MA "https://github.com/phubbard/tgn-whisperer/tree/main/episode_makefile"
    click ARCH "https://github.com/phubbard/tgn-whisperer/tree/main/archive"

    %% Styles
    classDef ingestion fill:#fde2e2,stroke:#f66,stroke-width:2px;
    classDef processing fill:#e2f7e2,stroke:#2a9d8f,stroke-width:2px;
    classDef orchestration fill:#e0e0e0,stroke:#333,stroke-width:2px;
    classDef siteGen fill:#e2e2ff,stroke:#4a90e2,stroke-width:2px;
    classDef external fill:#fff2cc,stroke:#e6b800,stroke-width:2px;
```
