# jaypyles/scraperr architecture

> Architecture diagram of the GitHub repository jaypyles/scraperr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jaypyles/scraperr
- Repository: https://github.com/jaypyles/scraperr

## Overview

##### Project type and purpose
- This is a full-stack, self-hosted web-scraping management application (“Scraperr”).
- Frontend: Next.js + TailwindCSS (React-based UI, form submission, results tables).
- Backend: FastAPI (REST API endpoints, authentication, job scheduling).
- Database: MongoDB (job metadata, results, schemas via Pydantic).
- Worker processes: Python scripts that perform scraping, media downloads, post-job notifications.
- Orchestration: Docker (compose & standalone), Kubernetes (Helm chart), cron scheduling, CI/CD via GitHub Actions.

##### File structure to infer architectural components
- Top-level
- api/backend → FastAPI code (routers, auth, database setup, job logic)
- src/pages & src/components → Next.js pages & React components
- docker/docker-files & docker-compose.yml → container definitions for frontend/backend
- helm/ → Helm chart for k8s deployment
- cypress/ → end-to-end tests against the running app
- worker modules under api/backend/worker → scraping workers, job dispatch, notification plugins
- api/backend/job/cron_scheduling → cron job definitions

##### README for technical stack & patterns
- FastAPI + MongoDB + Next.js + TailwindCSS
- Self-hosted with Docker or Helm
- Features hint at a client-server pattern plus a worker queue for background tasks and cron microservice.

##### Final advice
- Every logical boundary (UI vs API vs workers).
- Data flows and dependencies are explicit.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External User
    User["User’s Browser"]:::external

    %% Frontend Layer
    subgraph "Frontend Layer"
        direction TB
        NextJsUI["Next.js Frontend"]:::frontend
    end

    %% Backend/API Layer
    subgraph "Backend Layer"
        direction TB
        FastAPI["FastAPI Backend Service"]:::backend
    end

    %% Database Layer
    subgraph "Database Layer"
        direction TB
        MongoDB["MongoDB Database"]:::database
    end

    %% Scheduler & Workers
    subgraph "Background Processing" 
        direction TB
        Scheduler["Job Scheduler (cron)"]:::workers
        WorkerPool["Worker Pool"]:::workers
    end

    %% Notification & External
    subgraph "External Integrations"
        direction TB
        Notifications["Notification Services"]:::external
        Targets["Target Websites"]:::external
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        DockerCompose["Docker Compose"]:::infra
        HelmChart["Kubernetes Helm Chart"]:::infra
        CI_CD["CI/CD Workflows"]:::infra
        E2E["Cypress E2E Tests"]:::infra
    end

    %% Relationships
    User -->|"HTTPS: UI" | NextJsUI
    NextJsUI -->|"REST calls"| FastAPI
    FastAPI -->|"Read/Write"| MongoDB
    FastAPI -->|"Register jobs"| Scheduler
    Scheduler -->|"Trigger tasks"| WorkerPool
    WorkerPool -->|"Scrape HTTP"| Targets
    WorkerPool -->|"Store results"| MongoDB
    WorkerPool -->|"Notify"| Notifications

    %% Click Events - Frontend
    click NextJsUI "https://github.com/jaypyles/scraperr/tree/master/src/pages"
    click NextJsUI "https://github.com/jaypyles/scraperr/tree/master/src/components"
    click NextJsUI "https://github.com/jaypyles/scraperr/tree/master/docker/frontend/Dockerfile"

    %% Click Events - Backend
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/app.py"
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/routers/job_router.py"
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/routers/stats_router.py"
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/auth/auth_router.py"
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/ai/ai_router.py"
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/auth/auth_utils.py"
    click FastAPI "https://github.com/jaypyles/scraperr/blob/master/api/backend/scheduler.py"

    %% Click Events - Database
    click MongoDB "https://github.com/jaypyles/scraperr/blob/master/api/backend/database/startup.py"
    click MongoDB "https://github.com/jaypyles/scraperr/blob/master/api/backend/database/common.py"
    click MongoDB "https://github.com/jaypyles/scraperr/blob/master/api/backend/database/queries/queries.py"
    click MongoDB "https://github.com/jaypyles/scraperr/blob/master/api/backend/database/schema/schema.py"

    %% Click Events - Scheduler
    click Scheduler "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/cron_scheduling/cron_scheduling.py"
    click Scheduler "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/job.py"

    %% Click Events - Workers
    click WorkerPool "https://github.com/jaypyles/scraperr/blob/master/api/backend/worker/job_worker.py"
    click WorkerPool "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/scraping/scraping_utils.py"
    click WorkerPool "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/scraping/collect_media.py"
    click WorkerPool "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/site_mapping/site_mapping.py"
    click WorkerPool "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/models/job_options.py"
    click WorkerPool "https://github.com/jaypyles/scraperr/blob/master/api/backend/job/models/site_map.py"

    %% Click Events - Notifications
    click Notifications "https://github.com/jaypyles/scraperr/tree/master/api/backend/worker/post_job_complete"
    click Notifications "https://github.com/jaypyles/scraperr/blob/master/api/backend/worker/post_job_complete/discord_notification.py"
    click Notifications "https://github.com/jaypyles/scraperr/blob/master/api/backend/worker/post_job_complete/email_notifcation.py"
    click Notifications "https://github.com/jaypyles/scraperr/blob/master/api/backend/worker/post_job_complete/post_job_complete.py"

    %% Click Events - Infrastructure
    click DockerCompose "https://github.com/jaypyles/scraperr/blob/master/docker-compose.yml"
    click DockerCompose "https://github.com/jaypyles/scraperr/blob/master/docker-compose.dev.yml"
    click DockerCompose "https://github.com/jaypyles/scraperr/tree/master/docker/api/Dockerfile"
    click HelmChart "https://github.com/jaypyles/scraperr/blob/master/helm/Chart.yaml"
    click HelmChart "https://github.com/jaypyles/scraperr/blob/master/helm/values.yaml"
    click HelmChart "https://github.com/jaypyles/scraperr/blob/master/helm/templates/deployment.yaml"
    click HelmChart "https://github.com/jaypyles/scraperr/blob/master/helm/templates/service.yaml"
    click E2E "https://github.com/jaypyles/scraperr/blob/master/cypress.config.ts"
    click E2E "https://github.com/jaypyles/scraperr/tree/master/cypress/e2e"
    click E2E "https://github.com/jaypyles/scraperr/tree/master/cypress/fixtures"
    click E2E "https://github.com/jaypyles/scraperr/tree/master/cypress/support"
    click CI_CD "https://github.com/jaypyles/scraperr/blob/master/.github/workflows/docker-image.yml"
    click CI_CD "https://github.com/jaypyles/scraperr/blob/master/.github/workflows/unit-tests.yml"
    click CI_CD "https://github.com/jaypyles/scraperr/tree/master/.github/actions/push-to-helm"
    click CI_CD "https://github.com/jaypyles/scraperr/tree/master/.github/actions/run-cypress-tests"

    %% Styles
    classDef frontend fill:#a4f9c8,stroke:#067d68,stroke-width:1px
    classDef backend fill:#a0c4ff,stroke:#1f3a93,stroke-width:1px
    classDef database fill:#d3b8ff,stroke:#5f27cd,stroke-width:1px
    classDef workers fill:#ffd8a8,stroke:#ee7600,stroke-width:1px
    classDef external fill:#dfe6e9,stroke:#636e72,stroke-width:1px
    classDef infra fill:#feca57,stroke:#d35400,stroke-width:1px
```
