How jayu234/auto-scraper works
Project’s Main Components
• Frontend/UI: Represented by the “app” directory, this includes pages for authentication (sign-in/sign-up), dashboard (billing, credentials, workflows), workflow editors, and visualizations (using ReactFlow for the drag-and-drop editor). Also include the shared “components” folder which holds reusable UI elements.
• API/Backend: The “app/api” routes and the “actions” folder contain serverless functions and API endpoints for all core functionalities: billing, credentials, dashboard analytics, and workflow management. Show that these endpoints interact with both the frontend and the back end.
• Workflow Engine & Execution Modules: The “lib/workflow” folder has subdirectories for creating flows, execution plans, executors, and tasks. This set of modules implements the business logic for executing scraping workflows (leveraging Puppeteer) and managing intermediate representations.
• Database: Prisma is used for database interactions, with “prisma/schema.prisma” and migration folders indicating a PostgreSQL backend for storing everything from user details to workflow definitions, execution logs, and billing records.
• External Services Integration:
• Authentication with Clerk.
• Payment processing with Stripe (as well as webhooks under “app/api/webhooks/stripe” and related logic in “lib/stripe”).
• AI capabilities for smart automation (mentioned in README, possibly integrated with custom modules).
• Scheduling/Automation: Cron jobs and scheduled tasks are evident (e.g., “app/api/workflows/cron”) so include this as an automated task scheduler component.
Relationships and Interactions
• The Frontend communicates with the Backend via Next.js API routes and actions.
• API endpoints serve as intermediaries for:
• Initiating workflow executions (triggering the workflow engine modules and Puppeteer instances).
• Managing user credentials, billing, and dashboard stats.
• The Workflow Engine (in “lib/workflow”) is integrated with the API endpoints to execute and schedule workflows.
• The database (using Prisma) is accessed by multiple components for persisting user data, workflow configurations, billing histories, and logs.
• External webhooks and integration points ensure the system can react to Stripe events and use Clerk for secure authentication.
Architectural Patterns & Key Design Principles
• Separation of Concerns – Clearly divide the system into the UI (frontend), business logic (API endpoints, actions, workflow engine), and persistence (database layer).
• Modular Design – The “actions” and “lib/workflow” folders show a modular approach to supporting various features (billing, workflows, credentials) and allow for independent evolution of parts.
• Event-Driven Communication – Especially with webhooks from Stripe and scheduled cron endpoints running automated tasks.
• Extensibility – The design supports pluggable executors and tasks (observed through registries in the workflow modules), which allows new types of scraping actions to be added.
Technologies and Frameworks
• Technologies: Next.js, TypeScript, Prisma, PostgreSQL, Clerk, Stripe, Puppeteer.
• UI libraries: shadcn.ui and ReactFlow for interactive workflow design.
Summary & Final Touches
• Every significant architectural element – from authentication, billing, workflow management, to execution and external integrations.
• Avoid over-complication: focus primarily on the boundaries between components and the main data flows.