How ling-jun/react-fastapi-scraping-dashboard works
Project Type and Purpose
• This is a full-stack web application.
• Frontend: A React SPA in the hub-react folder.
• Backend: A FastAPI service (with Jinja2 templates fallback) plus Celery workers for asynchronous tasks.
• Purpose: Crawl/scrape URLs, track content changes (including a semantic‐diff step using a pretrained Sentence-Transformer), store/persist results via GitHub or Azure Blob Storage, and send email notifications.
Main Components
a. Frontend Service (“hub-react”)
• React application served via its own dev server or bundled into static assets.
• Fetches data and invokes operations via HTTP calls to the FastAPI backend.
b. Backend API Service (“backend/app.py” + routers)
• FastAPI application exposing REST endpoints (e.g., /scrape_again, /semantic_change, /send_email).
• Renders Jinja2 templates if used in “landing” endpoint.
• Reads config.ini for GitHub, Gmail credentials.
c. Asynchronous Task Layer (Celery)
• celery_app.py defines the broker & result backend.
• celery_tasks.py implements long-running jobs: scraping, semantic diff, email sending.
• Message broker (e.g., Redis or RabbitMQ) sits between FastAPI and worker.
d. Machine‐Learning Artifact
• Pretrained SentenceTransformer model under backend/artifacts/... used to compute semantic changes.
e. Data Persistence & External Integrations
• GitHub Remote: CSV file in a GitHub repo via utils/github_remote.py.
• Azure Blob Storage via utils/az_blob_store.py (optional alternative).
• SMTP/Gmail for outbound emails via utils/email_handling.py.
f. Static Assets & Templates
• Static JS/CSS in backend/static and HTML templates in backend/templates for optional FastAPI-hosted UI.
Relationships & Data Flows
• User (Browser) → React SPA → HTTP → FastAPI API
• FastAPI API → Immediate Response or → enqueue Celery Task → Message Broker
• Celery Worker → Pull Task →
• Invoke Scraper / Diff modules (utils/scrapeNsearchUtils.py, utils/detect_significant_changes.py)
• Load ML model from local artifacts
• Update data store (GitHub CSV or Azure Blob)
• Optionally send emails
• FastAPI API reads/writes same data store and serves updated dashboards.
Architectural Patterns & Design Principles
• Separation of concerns:
• Presentation (React or Jinja templates)
• API layer (FastAPI routers)
• Business logic (utils modules)
• Asynchronous processing (Celery)
• Modular directory structure:
• routers/ as controllers, utils/ as services, artifacts/ as ML models
• Config-driven credentials and external URLs via config.ini.
• Layer 2 (API): FastAPI container. Group its routers and template engine.
• Layer 3 (Async): Celery worker container + broker container.
• Layer 4 (Storage & External): GitHub API, Azure Blob Storage, Gmail SMTP.
• ML model artifact as an internal component of the Celery worker or API.
3. Inside FastAPI block, show routers → utils modules.
7. Optionally, show FastAPI reading from same GitHub/Blob to render dashboards.