# ling-jun/react-fastapi-scraping-dashboard architecture

> Architecture diagram of the GitHub repository ling-jun/react-fastapi-scraping-dashboard, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ling-jun/react-fastapi-scraping-dashboard
- Repository: https://github.com/ling-jun/react-fastapi-scraping-dashboard

## Overview

##### Project Type and Purpose
- This is a full-stack web application.
- Frontend: A React SPA in the `hub-react` folder.
- Backend: A FastAPI service (with Jinja2 templates fallback) plus Celery workers for asynchronous tasks.
- Purpose: Crawl/scrape URLs, track content changes (including a semantic‐diff step using a pretrained Sentence-Transformer), store/persist results via GitHub or Azure Blob Storage, and send email notifications.

##### Main Components
a. Frontend Service (“hub-react”)
- React application served via its own dev server or bundled into static assets.
- Fetches data and invokes operations via HTTP calls to the FastAPI backend.
b. Backend API Service (“backend/app.py” + routers)
- FastAPI application exposing REST endpoints (e.g., `/scrape_again`, `/semantic_change`, `/send_email`).
- Renders Jinja2 templates if used in “landing” endpoint.
- Reads `config.ini` for GitHub, Gmail credentials.
c. Asynchronous Task Layer (Celery)
- `celery_app.py` defines the broker & result backend.
- `celery_tasks.py` implements long-running jobs: scraping, semantic diff, email sending.
- Message broker (e.g., Redis or RabbitMQ) sits between FastAPI and worker.
d. Machine‐Learning Artifact
- Pretrained SentenceTransformer model under `backend/artifacts/...` used to compute semantic changes.
e. Data Persistence & External Integrations
- GitHub Remote: CSV file in a GitHub repo via `utils/github_remote.py`.
- Azure Blob Storage via `utils/az_blob_store.py` (optional alternative).
- SMTP/Gmail for outbound emails via `utils/email_handling.py`.
f. Static Assets & Templates
- Static JS/CSS in `backend/static` and HTML templates in `backend/templates` for optional FastAPI-hosted UI.

##### Relationships & Data Flows
- User (Browser) → React SPA → HTTP → FastAPI API
- FastAPI API → Immediate Response or → enqueue Celery Task → Message Broker
- Celery Worker → Pull Task →
- Invoke Scraper / Diff modules (`utils/scrapeNsearchUtils.py`, `utils/detect_significant_changes.py`)
- Load ML model from local artifacts
- Update data store (GitHub CSV or Azure Blob)
- Optionally send emails
- FastAPI API reads/writes same data store and serves updated dashboards.

##### Architectural Patterns & Design Principles
- Separation of concerns:
- Presentation (React or Jinja templates)
- API layer (FastAPI routers)
- Business logic (utils modules)
- Asynchronous processing (Celery)
- Modular directory structure:
- `routers/` as controllers, `utils/` as services, `artifacts/` as ML models
- Config-driven credentials and external URLs via `config.ini`.

- Layer 2 (API): FastAPI container. Group its routers and template engine.
- Layer 3 (Async): Celery worker container + broker container.
- Layer 4 (Storage & External): GitHub API, Azure Blob Storage, Gmail SMTP.
- ML model artifact as an internal component of the Celery worker or API.

3. Inside FastAPI block, show routers → utils modules.
7. Optionally, show FastAPI reading from same GitHub/Blob to render dashboards.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    User["Browser / User"]:::external
    User --> React

    %% Frontend Layer
    subgraph "Frontend (React SPA)" 
        direction TB
        React["React SPA"]:::frontend
        React --> MainTS
        React --> AppTS
        React --> DashboardTable
        React --> FetchData
    end

    MainTS["src/main.tsx"]:::frontend
    AppTS["src/App.tsx"]:::frontend
    DashboardTable["components/DashboardTable.tsx"]:::frontend
    FetchData["services/fetchData.ts"]:::frontend

    %% API Layer
    React -->|HTTP /scrape_again, /semantic_change, /send_email| API
    subgraph "Backend (FastAPI Service)" 
        direction TB
        API["FastAPI App (app.py)"]:::api
        Config["utils/config.py"]:::api
        subgraph "Routers"
            direction TB
            SemanticRouter["semantic_change.py"]:::api
            EmailRouter["send_email.py"]:::api
            LandingRouter["landing.py"]:::api
        end
        API --> Routers
        API --> Config
        subgraph "Static & Templates"
            direction TB
            StaticAssets["backend/static/"]:::api
            Templates["backend/templates/"]:::api
        end
        API --> StaticAssets
        API --> Templates
    end

    %% Async Layer
    API -->|Enqueue Task| Broker
    Broker["Message Broker\n(Redis/RabbitMQ)"]:::external

    subgraph "Celery Worker" 
        direction TB
        CeleryApp["celery_app.py"]:::worker
        CeleryTasks["celery_tasks.py"]:::worker
        Artifact["Sentence-Transformer Model\nartifacts/..."]:::worker
    end

    Broker -->|Deliver Task| CeleryApp
    CeleryApp --> CeleryTasks
    CeleryTasks --> Artifact

    %% Business Logic Utilities
    subgraph "Business Logic (utils)"
        direction TB
        ScrapeUtils["scrapeNsearchUtils.py"]:::api
        DiffUtils["detect_significant_changes.py"]:::api
        Tracker["change_tracker.py"]:::api
    end

    CeleryTasks --> ScrapeUtils
    CeleryTasks --> DiffUtils
    CeleryTasks --> Tracker
    API --> ScrapeUtils
    API --> DiffUtils
    API --> Tracker

    %% External Integrations
    subgraph "External Services" 
        direction TB
        GitHub["GitHub Remote (CSV)"]:::external
        AzureBlob["Azure Blob Storage"]:::external
        Gmail["SMTP / Gmail"]:::external
    end

    CeleryTasks --- GitHub
    CeleryTasks --- AzureBlob
    CeleryTasks --- Gmail
    API -->|Read/Write| GitHub
    API -->|Read/Write| AzureBlob

    %% Click Events
    click React "https://github.com/ling-jun/react-fastapi-scraping-dashboard/tree/main/hub-react/"
    click MainTS "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/hub-react/src/main.tsx"
    click AppTS "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/hub-react/src/App.tsx"
    click DashboardTable "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/hub-react/src/components/DashboardTable.tsx"
    click FetchData "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/hub-react/src/services/fetchData.ts"
    click API "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/app.py"
    click SemanticRouter "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/routers/semantic_change.py"
    click EmailRouter "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/routers/send_email.py"
    click LandingRouter "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/routers/landing.py"
    click CeleryApp "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/celery_app.py"
    click CeleryTasks "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/celery_tasks.py"
    click Artifact "https://github.com/ling-jun/react-fastapi-scraping-dashboard/tree/main/backend/artifacts/retrained_paraphrase-MiniLM-L6-v2/"
    click GitHub "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/github_remote.py"
    click AzureBlob "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/az_blob_store.py"
    click Gmail "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/email_handling.py"
    click ScrapeUtils "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/scrapeNsearchUtils.py"
    click DiffUtils "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/detect_significant_changes.py"
    click Tracker "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/change_tracker.py"
    click Config "https://github.com/ling-jun/react-fastapi-scraping-dashboard/blob/main/backend/utils/config.py"
    click StaticAssets "https://github.com/ling-jun/react-fastapi-scraping-dashboard/tree/main/backend/static/"
    click Templates "https://github.com/ling-jun/react-fastapi-scraping-dashboard/tree/main/backend/templates/"

    %% Styles
    classDef frontend fill:#E0F7FA,stroke:#006064,color:#000;
    classDef api fill:#E8F5E9,stroke:#1B5E20,color:#000;
    classDef worker fill:#FFF3E0,stroke:#E65100,color:#000;
    classDef external fill:#ECEFF1,stroke:#607D8B,color:#000;
```
