# jayu234/auto-scraper architecture

> Architecture diagram of the GitHub repository jayu234/auto-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jayu234/auto-scraper
- Repository: https://github.com/jayu234/auto-scraper

## Overview

##### Project’s Main Components
- Frontend/UI: Represented by the “app” directory, this includes pages for authentication (sign-in/sign-up), dashboard (billing, credentials, workflows), workflow editors, and visualizations (using ReactFlow for the drag-and-drop editor). Also include the shared “components” folder which holds reusable UI elements.
- API/Backend: The “app/api” routes and the “actions” folder contain serverless functions and API endpoints for all core functionalities: billing, credentials, dashboard analytics, and workflow management. Show that these endpoints interact with both the frontend and the back end.
- Workflow Engine & Execution Modules: The “lib/workflow” folder has subdirectories for creating flows, execution plans, executors, and tasks. This set of modules implements the business logic for executing scraping workflows (leveraging Puppeteer) and managing intermediate representations.
- Database: Prisma is used for database interactions, with “prisma/schema.prisma” and migration folders indicating a PostgreSQL backend for storing everything from user details to workflow definitions, execution logs, and billing records.
- External Services Integration:
- Authentication with Clerk.
- Payment processing with Stripe (as well as webhooks under “app/api/webhooks/stripe” and related logic in “lib/stripe”).
- AI capabilities for smart automation (mentioned in README, possibly integrated with custom modules).
- Scheduling/Automation: Cron jobs and scheduled tasks are evident (e.g., “app/api/workflows/cron”) so include this as an automated task scheduler component.

##### Relationships and Interactions
- The Frontend communicates with the Backend via Next.js API routes and actions.
- API endpoints serve as intermediaries for:
- Initiating workflow executions (triggering the workflow engine modules and Puppeteer instances).
- Managing user credentials, billing, and dashboard stats.
- The Workflow Engine (in “lib/workflow”) is integrated with the API endpoints to execute and schedule workflows.
- The database (using Prisma) is accessed by multiple components for persisting user data, workflow configurations, billing histories, and logs.
- External webhooks and integration points ensure the system can react to Stripe events and use Clerk for secure authentication.

##### Architectural Patterns & Key Design Principles
- Separation of Concerns – Clearly divide the system into the UI (frontend), business logic (API endpoints, actions, workflow engine), and persistence (database layer).
- Modular Design – The “actions” and “lib/workflow” folders show a modular approach to supporting various features (billing, workflows, credentials) and allow for independent evolution of parts.
- Event-Driven Communication – Especially with webhooks from Stripe and scheduled cron endpoints running automated tasks.
- Extensibility – The design supports pluggable executors and tasks (observed through registries in the workflow modules), which allows new types of scraping actions to be added.

##### Technologies and Frameworks
- Technologies: Next.js, TypeScript, Prisma, PostgreSQL, Clerk, Stripe, Puppeteer.
- UI libraries: shadcn.ui and ReactFlow for interactive workflow design.

##### Summary & Final Touches
- Every significant architectural element – from authentication, billing, workflow management, to execution and external integrations.
- Avoid over-complication: focus primarily on the boundaries between components and the main data flows.

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Layer
    U["User Browser"]:::user

    %% UI Layer
    subgraph "UI Layer"
        FUI["Frontend UI (Next.js + ReactFlow)"]:::frontend
        SUI["Shared UI Components"]:::frontend
    end

    %% API & Business Logic Layer
    subgraph "API & Business Logic"
        API["API Routes (Next.js API)"]:::backend
        ACT["Serverless Functions (Actions)"]:::backend
    end

    %% Workflow Engine Layer
    subgraph "Workflow Engine"
        WE["Workflow Engine Modules"]:::workflow
    end

    %% Data Layer
    DB["Database (PostgreSQL via Prisma)"]:::database

    %% External Services
    subgraph "External Services"
        AUTH["Authentication (Clerk)"]:::external
        PAY["Payment Integration (Stripe)"]:::external
    end

    %% Scheduler
    SCH["Scheduler/Cron Jobs"]:::scheduler

    %% Data flow connections
    U -->|"accesses"| FUI
    FUI -->|"calls"| API
    FUI -->|"uses"| SUI
    API -->|"triggers"| ACT
    API -->|"initiates"| WE
    ACT -->|"persists data in"| DB
    WE -->|"stores results in"| DB
    AUTH -->|"authenticates"| FUI
    AUTH -->|"secures"| API
    PAY -->|"sendsWebhookTo"| API
    SCH -->|"schedules"| WE

    %% Click Events for Frontend/UI
    click FUI "https://github.com/jayu234/auto-scraper/tree/main/app/"
    click SUI "https://github.com/jayu234/auto-scraper/tree/main/components/"
    
    %% Click Events for API/Backend
    click API "https://github.com/jayu234/auto-scraper/tree/main/app/api/"
    click ACT "https://github.com/jayu234/auto-scraper/tree/main/actions/"

    %% Click Event for Workflow Engine
    click WE "https://github.com/jayu234/auto-scraper/tree/main/lib/workflow/"

    %% Click Event for Database
    click DB "https://github.com/jayu234/auto-scraper/blob/main/prisma/schema.prisma"

    %% Click Events for External Services
    click AUTH "https://github.com/jayu234/auto-scraper/tree/main/app/(auth)/"
    click PAY "https://github.com/jayu234/auto-scraper/blob/main/app/api/webhooks/stripe/route.ts"

    %% Click Event for Scheduler/Cron Jobs
    click SCH "https://github.com/jayu234/auto-scraper/tree/main/app/api/workflows/cron/"

    %% Styling classes
    classDef user fill:#f9f,stroke:#333,stroke-width:2px;
    classDef frontend fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef backend fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef workflow fill:#ffa500,stroke:#000,stroke-width:2px;
    classDef database fill:#dda0dd,stroke:#000,stroke-width:2px;
    classDef external fill:#ffff99,stroke:#000,stroke-width:2px;
    classDef scheduler fill:#d3d3d3,stroke:#000,stroke-width:2px;
```
