# yatam-nikhila/scrapeflow architecture

> Architecture diagram of the GitHub repository yatam-nikhila/scrapeflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yatam-nikhila/scrapeflow
- Repository: https://github.com/yatam-nikhila/scrapeflow

## Overview

##### Project type and purpose
- This is a full-stack SaaS application (“ScrapeFlow”) for building, scheduling, executing, and monitoring web-scraping/browser-automation workflows.
- Frontend: Next.js App Router with a node-based visual editor (ReactFlow).
- Backend/API: Next.js API routes (TypeScript) serving CRUD, execution, scheduling, billing, and webhook endpoints.
- Worker: A separate BullMQ/Redis worker that runs Puppeteer-driven “sandboxRunner” jobs.
- Data stores: PostgreSQL (via Prisma), Redis queue, plus external services (Clerk for auth, Stripe for payments, optional AI API).

##### Key directories and architectural patterns
- src/app & src/pages/api – Next.js handles both UI pages and API routes in one codebase.
- src/components & src/lib – UI components (Shadcn/Radix), utilities (crypto, validators, Stripe, Prisma).
- src/server/queue & src/server/runners – Asynchronous job producer (enqueue.ts) and consumer/worker (worker.ts + sandboxRunner.ts).
- prisma/schema.prisma – Defines relational model.
- Dockerfile.worker – Container image for the worker process.
- CI/CD: GitHub Actions (ci.yml).

##### README for insights
- Visual Workflow Editor, Cron scheduling, real-time status, credit-based billing, secure vault.
- Technology stack confirms Next.js, TypeScript, Tailwind, Prisma, Redis+BullMQ, Puppeteer, Clerk, Stripe, Cheerio, Zod, Vitest.

##### Final advice
- Essential components: UI, API, DB, Queue, Worker, External Integrations
- Both synchronous (API CRUD) and asynchronous (queue) flows
- Encryption/vault and scheduling as special sub-systems

## Diagram (Mermaid)

```mermaid
graph TB
  subgraph "Frontend/UI"
    Browser["Client Browser"]:::frontend
    NextApp["Next.js App Router"]:::frontend
    NodeEditor["Visual Workflow Editor"]:::frontend
    Dashboard["Dashboard & Cards"]:::frontend
    ExecUI["Execution History UI"]:::frontend
    BillingUI["Billing Top-up UI"]:::frontend
    VaultUI["Credential Vault UI"]:::frontend
    SharedUI["Shared UI Components"]:::frontend
  end

  subgraph "Next.js Server"
    API["API Gateway (/api/*)"]:::backend
    AuthM["Auth Middleware"]:::backend
  end

  subgraph "Data Layer"
    PG["PostgreSQL Database"]:::database
    Prisma["Prisma ORM"]:::backend
    Redis["Redis + BullMQ"]:::database
  end

  subgraph "Worker (Docker)"
    Worker["Queue Consumer Worker"]:::worker
    Sandbox["Sandbox Runner (Puppeteer)"]:::worker
  end

  subgraph "Utilities"
    StripeUtil["Stripe Utilities"]:::backend
    CryptoUtil["Crypto & Vault Utilities"]:::backend
    ValValidators["Zod Validators"]:::backend
  end

  Clerk["Clerk Auth Service"]:::external
  Stripe["Stripe Service"]:::external
  AI["AI Provider API"]:::external
  Webhook["User Webhook Endpoint"]:::external

  Browser --> NextApp
  NextApp --> API
  Browser -->|poll/subscribe| API
  Browser -->|Auth flows| Clerk
  API -->|verify session| Clerk

  API --> Prisma
  Prisma --> PG
  API --> CryptoUtil
  API --> StripeUtil
  API --> Redis
  API -->|enqueue job| Redis

  Redis --> Worker
  Worker --> Sandbox
  Sandbox --> PG
  Sandbox -->|callback| Webhook

  Browser -->|stripe checkout| Stripe
  Stripe --> API

  API -->|optional AI calls| AI
  Worker -->|optional AI calls| AI

  classDef frontend fill:#a2f5a2,stroke:#333,stroke-width:1px
  classDef backend fill:#a2c8f5,stroke:#333,stroke-width:1px
  classDef worker fill:#f5c78a,stroke:#333,stroke-width:1px
  classDef database fill:#d5a4f5,stroke:#333,stroke-width:1px
  classDef external fill:#f58a8a,stroke:#333,stroke-width:1px

  click NextApp "https://github.com/yatam-nikhila/scrapeflow/tree/main/src/app"
  click NodeEditor "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/components/NodeEditor.tsx"
  click Dashboard "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/components/Dashboard.tsx"
  click ExecUI "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/components/ExecutionHistory.tsx"
  click BillingUI "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/components/BillingTopup.tsx"
  click VaultUI "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/components/CredentialVault.tsx"
  click SharedUI "https://github.com/yatam-nikhila/scrapeflow/tree/main/src/components/ui/*"
  click API "https://github.com/yatam-nikhila/scrapeflow/tree/main/src/pages/api/*"
  click AuthM "https://github.com/yatam-nikhila/scrapeflow/blob/main/middleware.ts"
  click AuthM "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/lib/auth.ts"
  click Prisma "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/lib/prisma.ts"
  click PG "https://github.com/yatam-nikhila/scrapeflow/blob/main/prisma/schema.prisma"
  click Worker "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/server/queue/worker.ts"
  click Worker "https://github.com/yatam-nikhila/scrapeflow/blob/main/Dockerfile.worker"
  click Sandbox "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/server/runners/sandboxRunner.ts"
  click StripeUtil "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/lib/stripe.ts"
  click CryptoUtil "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/lib/crypto.ts"
  click ValValidators "https://github.com/yatam-nikhila/scrapeflow/blob/main/src/lib/validators.ts"
```
