How jaypyles/scraperr works
Project type and purpose
• This is a full-stack, self-hosted web-scraping management application (“Scraperr”).
• Frontend: Next.js + TailwindCSS (React-based UI, form submission, results tables).
• Backend: FastAPI (REST API endpoints, authentication, job scheduling).
• Database: MongoDB (job metadata, results, schemas via Pydantic).
• Worker processes: Python scripts that perform scraping, media downloads, post-job notifications.
• Orchestration: Docker (compose & standalone), Kubernetes (Helm chart), cron scheduling, CI/CD via GitHub Actions.
File structure to infer architectural components
• Top-level
• api/backend → FastAPI code (routers, auth, database setup, job logic)
• src/pages & src/components → Next.js pages & React components
• docker/docker-files & docker-compose.yml → container definitions for frontend/backend
• helm/ → Helm chart for k8s deployment
• cypress/ → end-to-end tests against the running app
• worker modules under api/backend/worker → scraping workers, job dispatch, notification plugins
• api/backend/job/cron_scheduling → cron job definitions
README for technical stack & patterns
• FastAPI + MongoDB + Next.js + TailwindCSS
• Self-hosted with Docker or Helm
• Features hint at a client-server pattern plus a worker queue for background tasks and cron microservice.
Final advice
• Every logical boundary (UI vs API vs workers).
• Data flows and dependencies are explicit.