How yatam-nikhila/scrapeflow works
Project type and purpose
• This is a full-stack SaaS application (“ScrapeFlow”) for building, scheduling, executing, and monitoring web-scraping/browser-automation workflows.
• Frontend: Next.js App Router with a node-based visual editor (ReactFlow).
• Backend/API: Next.js API routes (TypeScript) serving CRUD, execution, scheduling, billing, and webhook endpoints.
• Worker: A separate BullMQ/Redis worker that runs Puppeteer-driven “sandboxRunner” jobs.
• Data stores: PostgreSQL (via Prisma), Redis queue, plus external services (Clerk for auth, Stripe for payments, optional AI API).
Key directories and architectural patterns
• src/app & src/pages/api – Next.js handles both UI pages and API routes in one codebase.
• src/components & src/lib – UI components (Shadcn/Radix), utilities (crypto, validators, Stripe, Prisma).
• src/server/queue & src/server/runners – Asynchronous job producer (enqueue.ts) and consumer/worker (worker.ts + sandboxRunner.ts).
• prisma/schema.prisma – Defines relational model.
• Dockerfile.worker – Container image for the worker process.
• CI/CD: GitHub Actions (ci.yml).
README for insights
• Visual Workflow Editor, Cron scheduling, real-time status, credit-based billing, secure vault.
• Technology stack confirms Next.js, TypeScript, Tailwind, Prisma, Redis+BullMQ, Puppeteer, Clerk, Stripe, Cheerio, Zod, Vitest.
Final advice
• Essential components: UI, API, DB, Queue, Worker, External Integrations
• Both synchronous (API CRUD) and asynchronous (queue) flows
• Encryption/vault and scheduling as special sub-systems