How alexanderjuxoncobb/selenium-hacker-news-scraper works
Project Type & Purpose
• This is a full-stack, production-ready web platform that scrapes Hacker News, runs a cost-optimized AI pipeline for summarization & insights, manages multi-user interest profiles, serves a FastAPI dashboard, and delivers daily personalized email digests.
File Structure Analysis
• Top-level scripts & modules:
• multi_user_scraper.py, scraper.py, enhanced_scraper.py – data ingestion
• ai_pipeline.py, actionable_insights.py, interest_learner.py – AI processing
• email_sender.py – email delivery
• scheduler.py, railway_cron.py, railway_scheduler.py – job scheduling
• dashboard/ – FastAPI web application (app.py, database models, templates, static assets)
• migration & utility scripts (migrate_*.py, fix_*.py) – DB migrations & repairs
• .ai_cache/ – local cache for summaries
• outputs/ – scrape JSON + generated markdown digests
Key README Insights
• Architecture section in README outlines:
• Hybrid AI (local embeddings + selective OpenAI)
• Scheduler triggers daily runs at 8:30 AM London time (Railway Cron + local schedule)
• PostgreSQL (prod) / SQLite (local) database storing users, interests, story records, analytics
• FastAPI + Jinja2 dashboard for user/admin flows
• Resend.com integration for email digests
Components
a. Data Ingestion Layer
• scheduler.py / railway_cron.py → triggers scraper scripts
• scraper modules (multi_user_scraper.py calls enhanced_scraper.py)
• External: Hacker News website (via Selenium & HTTP requests)
b. AI Processing Layer
• ai_pipeline.py (local embeddings with sentence-transformers)
• OpenAI API (selective refinement & summaries)
• actionable_insights.py for business intelligence extraction
• .ai_cache directory for caching article summaries
c. Persistence Layer
• PostgreSQL (prod) / SQLite (local)
• database tables: users, interests, stories, summaries, analytics, migrations
• dashboard/database.py manages ORM operations (psycopg2 or sqlite3)
d. Web Application Layer
• FastAPI app.py serving:
• Public routes: /setup, /landing, /interests, /dashboard
• Admin routes: /admin, /admin_users, /admin_analytics
• Jinja2 templates + static CSS/JS
• Uvicorn ASGI server
e. Delivery Layer
• email_sender.py composes & sends daily digests via Resend.com API
• outputs/digests stores Markdown files
Interactions & Data Flows
1. Scheduler (Railway Cron or scheduler.py) → triggers multi_user_scraper.py
2. multi_user_scraper.py loops through active users:
→ calls enhanced_scraper.py (Selenium & requests) → raw JSON
→ passes JSON to ai_pipeline.py → embeddings + selective OpenAI calls → summaries & analytics
→ actionable_insights.py extracts market signals
→ writes results to Database
3. After processing all users, scheduler invokes email_sender.py:
→ queries DB for each user’s relevant stories & summaries
→ calls Resend.com API to send personalized Markdown/HTML digest
4. FastAPI dashboard (app.py) interacts with the same DB:
→ user onboarding, interest management, admin analytics
→ UI requests → database operations → template rendering
Architectural Patterns & Principles
• Modular design: separate scripts for ingestion, AI processing, delivery, web UI
• Hybrid AI pattern: local processing first, cloud fallback for edge cases
• Caching: avoid redundant OpenAI calls via .ai_cache
• Cron/Scheduler design for daily batch jobs
• MVC-style pattern in dashboard: models (database.py), views (templates), controllers (FastAPI routes)