# alexanderjuxoncobb/selenium-hacker-news-scraper architecture

> Architecture diagram of the GitHub repository alexanderjuxoncobb/selenium-hacker-news-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alexanderjuxoncobb/selenium-hacker-news-scraper
- Repository: https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper

## Overview

##### Project Type & Purpose
- This is a full-stack, production-ready web platform that scrapes Hacker News, runs a cost-optimized AI pipeline for summarization & insights, manages multi-user interest profiles, serves a FastAPI dashboard, and delivers daily personalized email digests.

##### File Structure Analysis
- Top-level scripts & modules:
- multi_user_scraper.py, scraper.py, enhanced_scraper.py – data ingestion
- ai_pipeline.py, actionable_insights.py, interest_learner.py – AI processing
- email_sender.py – email delivery
- scheduler.py, railway_cron.py, railway_scheduler.py – job scheduling
- dashboard/ – FastAPI web application (app.py, database models, templates, static assets)
- migration & utility scripts (migrate_*.py, fix_*.py) – DB migrations & repairs
- .ai_cache/ – local cache for summaries
- outputs/ – scrape JSON + generated markdown digests

##### Key README Insights
- Architecture section in README outlines:
- Hybrid AI (local embeddings + selective OpenAI)
- Scheduler triggers daily runs at 8:30 AM London time (Railway Cron + local schedule)
- PostgreSQL (prod) / SQLite (local) database storing users, interests, story records, analytics
- FastAPI + Jinja2 dashboard for user/admin flows
- Resend.com integration for email digests

##### Components
a. Data Ingestion Layer
- scheduler.py / railway_cron.py → triggers scraper scripts
- scraper modules (multi_user_scraper.py calls enhanced_scraper.py)
- External: Hacker News website (via Selenium & HTTP requests)
b. AI Processing Layer
- ai_pipeline.py (local embeddings with sentence-transformers)
- OpenAI API (selective refinement & summaries)
- actionable_insights.py for business intelligence extraction
- .ai_cache directory for caching article summaries
c. Persistence Layer
- PostgreSQL (prod) / SQLite (local)
- database tables: users, interests, stories, summaries, analytics, migrations
- dashboard/database.py manages ORM operations (psycopg2 or sqlite3)
d. Web Application Layer
- FastAPI app.py serving:
- Public routes: /setup, /landing, /interests, /dashboard
- Admin routes: /admin, /admin_users, /admin_analytics
- Jinja2 templates + static CSS/JS
- Uvicorn ASGI server
e. Delivery Layer
- email_sender.py composes & sends daily digests via Resend.com API
- outputs/digests stores Markdown files

##### Interactions & Data Flows
1. Scheduler (Railway Cron or scheduler.py) → triggers multi_user_scraper.py
2. multi_user_scraper.py loops through active users:
→ calls enhanced_scraper.py (Selenium & requests) → raw JSON
→ passes JSON to ai_pipeline.py → embeddings + selective OpenAI calls → summaries & analytics
→ actionable_insights.py extracts market signals
→ writes results to Database
3. After processing all users, scheduler invokes email_sender.py:
→ queries DB for each user’s relevant stories & summaries
→ calls Resend.com API to send personalized Markdown/HTML digest
4. FastAPI dashboard (app.py) interacts with the same DB:
→ user onboarding, interest management, admin analytics
→ UI requests → database operations → template rendering

##### Architectural Patterns & Principles
- Modular design: separate scripts for ingestion, AI processing, delivery, web UI
- Hybrid AI pattern: local processing first, cloud fallback for edge cases
- Caching: avoid redundant OpenAI calls via .ai_cache
- Cron/Scheduler design for daily batch jobs
- MVC-style pattern in dashboard: models (database.py), views (templates), controllers (FastAPI routes)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer" 
        direction TB
        Scheduler["Scheduler\n(schedule / Railway Cron)"]:::orange
        MultiScraper["multi_user_scraper.py\n(Selenium, Requests)"]:::orange
        EnhancedScraper["enhanced_scraper.py\n(BeautifulSoup4)"]:::orange
    end

    %% AI Processing Layer
    subgraph "AI Processing Layer"
        direction TB
        AIPipeline["ai_pipeline.py\n(sentence-transformers)"]:::green
        ActionInsights["actionable_insights.py\n(BI Extraction)"]:::green
        InterestLearner["interest_learner.py"]:::green
        AICache[".ai_cache/\n(article_summaries.pkl)"]:::gray
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        direction TB
        Database["PostgreSQL / SQLite\n(psycopg2 / sqlite3)"]:::db
        DatabaseMgmt["dashboard/database.py\n(ORM)"]:::db
    end

    %% Web Application Layer
    subgraph "Web Application Layer"
        direction TB
        WebApp["dashboard/app.py\n(FastAPI + Uvicorn + Jinja2)"]:::blue
        Templates["dashboard/templates/"]:::blue
        StaticCSS["dashboard/static/css/"]:::blue
        StaticJS["dashboard/static/js/"]:::blue
    end

    %% Delivery Layer
    subgraph "Delivery Layer"
        direction TB
        EmailSender["email_sender.py\n(Resend.com API)"]:::orange
        Digests["outputs/digests/"]:::gray
        ScrapesOut["outputs/scrapes/"]:::gray
    end

    %% External Services
    HackerNews["Hacker News"]:::ext
    OpenAI["OpenAI API"]:::ext
    Resend["Resend.com"]:::ext
    User["End User"]:::ext

    %% Flows
    Scheduler -->|"triggers"| MultiScraper
    MultiScraper -->|POST raw JSON| AIPipeline
    AIPipeline -->|caches/reads summaries| AICache
    AIPipeline -->|GET embeddings & summaries| OpenAI
    AIPipeline -->|processed data| ActionInsights
    ActionInsights -->|analytics| Database
    AIPipeline -->|INSERT summaries| Database
    MultiScraper -->|"scrapes"| HackerNews
    Database -->|queried by| WebApp
    WebApp -->|renders| Templates
    WebApp -->|loads| StaticCSS
    WebApp -->|loads| StaticJS
    WebApp -->|serves to| User
    Database -->|reads stories| EmailSender
    EmailSender -->|POST email| Resend
    Resend -->|delivers| User
    MultiScraper -->|stores raw JSON| ScrapesOut
    EmailSender -->|stores markdown| Digests

    %% Click Events
    click Scheduler "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/scheduler.py"
    click Scheduler "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/railway_cron.py"
    click Scheduler "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/railway_scheduler.py"
    click MultiScraper "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/multi_user_scraper.py"
    click EnhancedScraper "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/enhanced_scraper.py"
    click AIPipeline "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/ai_pipeline.py"
    click ActionInsights "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/actionable_insights.py"
    click InterestLearner "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/interest_learner.py"
    click AICache "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/tree/main/.ai_cache/"
    click DatabaseMgmt "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/dashboard/database.py"
    click WebApp "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/dashboard/app.py"
    click Templates "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/tree/main/dashboard/templates/"
    click StaticCSS "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/tree/main/dashboard/static/css/"
    click StaticJS "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/tree/main/dashboard/static/js/"
    click EmailSender "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/blob/main/email_sender.py"
    click Digests "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/tree/main/outputs/digests/"
    click ScrapesOut "https://github.com/alexanderjuxoncobb/selenium-hacker-news-scraper/tree/main/outputs/scrapes/"

    %% Styles
    classDef orange fill:#FFE5B4,stroke:#E59400,color:#000000
    classDef green fill:#D4F4DD,stroke:#2E7D32,color:#000000
    classDef blue fill:#D0E8FF,stroke:#2071C3,color:#000000
    classDef gray fill:#E0E0E0,stroke:#757575,color:#000000
    classDef ext fill:#C8F0FF,stroke:#0091EA,color:#000000
    classDef db fill:#F0E6A8,stroke:#A67C00,color:#000000
```
