# jfcostello/article-summarizer architecture

> Architecture diagram of the GitHub repository jfcostello/article-summarizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jfcostello/article-summarizer
- Repository: https://github.com/jfcostello/article-summarizer

## Overview

##### Main Components

a) Core Processing Pipeline:
- URL Fetcher Service
- Scraper Service
- Summarizer Service
- Tagger Service

b) Infrastructure Components:
- Celery Worker & Beat (Task Scheduling)
- RabbitMQ (Message Broker)
- Supabase Database
- Multiple LLM Services (Groq, Claude, Gemini, TogetherAI, Replicate)

**[RSS Feeds] → [URL Fetcher] → [Scraper] → [Summarizer] → [Tagger] → [Supabase DB]**

##### Key Elements to Highlight

a) Task Management:
- Celery worker and beat scheduler
- Task queue flow
- Redundancy paths

b) Data Storage:
- Main Supabase tables
- Data flow between services and storage

c) External Services:
- LLM providers
- API connections
- Fallback relationships

##### Additional Details

Annotations for:
- Configuration management
- Logging and monitoring
- Error handling paths
- Task scheduling intervals

##### Specific Implementation Notes

- Redundancy manager's role in orchestrating multiple implementations
- How the task queue ensures sequential processing
- Relationship between interfaces and their implementations

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph RSS["RSS Layer"]
        RSS_FEEDS[RSS Feeds]
    end

    subgraph FETCH["URL Fetcher Layer"]
        URL_FETCHER_IF["URL Fetcher Interface"]:::interface
        URL_FETCHER["FeedParser Implementation"]:::implementation
        URL_UTILS["URL Fetch Utils"]:::utils
    end

    subgraph SCRAPE["Scraper Layer"]
        SCRAPER_IF["Scraper Interface"]:::interface
        SCRAPER["Puppeteer Implementation"]:::implementation
        SCRAPE_UTILS["Scraping Utils"]:::utils
    end

    subgraph SUMM["Summarizer Layer"]
        SUMM_IF["Summarizer Interface"]:::interface
        SUMM_UTILS["Summarizer Utils"]:::utils
        
        subgraph LLM_SUMM["LLM Implementations"]
            CLAUDE_SUMM["Claude Haiku"]:::llm
            GEMINI_SUMM["Gemini Flash"]:::llm
            GROQ_SUMM["Groq Llama"]:::llm
            REPLICATE_SUMM["Replicate Llama"]:::llm
            TOGETHER_SUMM["TogetherAI Llama"]:::llm
        end
    end

    subgraph TAG["Tagger Layer"]
        TAG_IF["Tagger Interface"]:::interface
        TAG_UTILS["Tagger Utils"]:::utils
        
        subgraph LLM_TAG["LLM Implementations"]
            CLAUDE_TAG["Claude Haiku"]:::llm
            GEMINI_TAG["Gemini Flash"]:::llm
            GROQ_TAG["Groq Llama"]:::llm
            REPLICATE_TAG["Replicate Llama"]:::llm
            TOGETHER_TAG["TogetherAI Llama"]:::llm
        end
    end

    subgraph INFRA["Infrastructure Layer"]
        CELERY["Celery Worker & Beat"]:::infra
        RABBITMQ["RabbitMQ"]:::infra
        SUPABASE["Supabase DB"]:::storage
        DB_UTILS["Database Utils"]:::utils
        LLM_UTILS["LLM Utils"]:::utils
        LOG_UTILS["Logging Utils"]:::utils
    end

    subgraph CONFIG["Configuration Layer"]
        CONFIG_FILE["Config File"]:::config
        CONFIG_LOADER["Config Loader"]:::config
    end

    subgraph REDUNDANCY["Redundancy Management"]
        RED_MANAGER["Redundancy Manager"]:::manager
    end

    subgraph MAIN["Main Pipeline"]
        MAIN_SCRIPT["Main Script"]:::main
    end

    %% Connections
    RSS_FEEDS --> URL_FETCHER
    URL_FETCHER_IF --> URL_FETCHER
    URL_FETCHER --> URL_UTILS
    URL_UTILS --> SUPABASE

    SCRAPER_IF --> SCRAPER
    SCRAPER --> SCRAPE_UTILS
    SCRAPE_UTILS --> SUPABASE

    SUMM_IF --> LLM_SUMM
    LLM_SUMM --> SUMM_UTILS
    SUMM_UTILS --> SUPABASE

    TAG_IF --> LLM_TAG
    LLM_TAG --> TAG_UTILS
    TAG_UTILS --> SUPABASE

    CELERY --> RABBITMQ
    RABBITMQ --> MAIN_SCRIPT
    
    RED_MANAGER --> LLM_SUMM
    RED_MANAGER --> LLM_TAG
    
    CONFIG_FILE --> CONFIG_LOADER
    CONFIG_LOADER --> MAIN_SCRIPT

    DB_UTILS --> SUPABASE
    LLM_UTILS --> LLM_SUMM
    LLM_UTILS --> LLM_TAG
    LOG_UTILS --> MAIN_SCRIPT

    %% Click Events
    click URL_FETCHER_IF "https://github.com/jfcostello/article-summarizer/blob/main/interfaces/url_fetcher.py"
    click URL_FETCHER "https://github.com/jfcostello/article-summarizer/blob/main/scripts/fetch_urls/fetch_urls_feedparser.py"
    click URL_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/url_fetch_utils.py"
    
    click SCRAPER_IF "https://github.com/jfcostello/article-summarizer/blob/main/interfaces/scraper.py"
    click SCRAPER "https://github.com/jfcostello/article-summarizer/blob/main/scripts/scraper/scrape_puppeteer.py"
    click SCRAPE_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/scraping_util.py"
    
    click SUMM_IF "https://github.com/jfcostello/article-summarizer/blob/main/interfaces/summarizer.py"
    click CLAUDE_SUMM "https://github.com/jfcostello/article-summarizer/blob/main/scripts/summarizer/summarizer_claude_haiku.py"
    click GEMINI_SUMM "https://github.com/jfcostello/article-summarizer/blob/main/scripts/summarizer/summarizer_gemini_flash.py"
    click GROQ_SUMM "https://github.com/jfcostello/article-summarizer/blob/main/scripts/summarizer/summarizer_groq_llama8b.py"
    click REPLICATE_SUMM "https://github.com/jfcostello/article-summarizer/blob/main/scripts/summarizer/summarizer_replicate_llama8b.py"
    click TOGETHER_SUMM "https://github.com/jfcostello/article-summarizer/blob/main/scripts/summarizer/summarizer_togetherai_llama8b.py"
    click SUMM_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/summarizer_utils.py"
    
    click TAG_IF "https://github.com/jfcostello/article-summarizer/blob/main/interfaces/tagger.py"
    click CLAUDE_TAG "https://github.com/jfcostello/article-summarizer/blob/main/scripts/tagging/tagging_claude_haiku.py"
    click GEMINI_TAG "https://github.com/jfcostello/article-summarizer/blob/main/scripts/tagging/tagging_gemini_flash.py"
    click GROQ_TAG "https://github.com/jfcostello/article-summarizer/blob/main/scripts/tagging/tagging_groq_llama8b.py"
    click REPLICATE_TAG "https://github.com/jfcostello/article-summarizer/blob/main/scripts/tagging/tagging_replicate_llama8b.py"
    click TOGETHER_TAG "https://github.com/jfcostello/article-summarizer/blob/main/scripts/tagging/tagging_togetherai_llama8b.py"
    click TAG_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/tagging_utils.py"
    
    click CELERY "https://github.com/jfcostello/article-summarizer/blob/main/task_management/celery_app.py"
    click RED_MANAGER "https://github.com/jfcostello/article-summarizer/blob/main/scripts/redundancy_manager.py"
    click CONFIG_FILE "https://github.com/jfcostello/article-summarizer/blob/main/config/config.yaml"
    click CONFIG_LOADER "https://github.com/jfcostello/article-summarizer/blob/main/config/config_loader.py"
    click DB_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/db_utils.py"
    click LLM_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/llm_utils.py"
    click LOG_UTILS "https://github.com/jfcostello/article-summarizer/blob/main/utils/logging_utils.py"
    click MAIN_SCRIPT "https://github.com/jfcostello/article-summarizer/blob/main/scripts/main.py"

    %% Styles
    classDef interface fill:#a8d1ff,stroke:#333
    classDef implementation fill:#90ee90,stroke:#333
    classDef utils fill:#ffb366,stroke:#333
    classDef llm fill:#c2f0c2,stroke:#333
    classDef infra fill:#d1d1e0,stroke:#333
    classDef storage fill:#ffff99,stroke:#333
    classDef config fill:#e6ccff,stroke:#333
    classDef manager fill:#ff9999,stroke:#333
    classDef main fill:#ff99cc,stroke:#333
```
