How tharuneshwar-s/firecrawl works
Project type and purpose
• Firecrawl is a mono-repo for a SaaS “web crawling + data extraction” platform. It provides:
• A Node/TypeScript REST API (apps/api) that handles crawl, scrape, map, search, extract endpoints.
• A headless-browser microservice (apps/playwright-service-ts) for JS-rendered sites.
• Redis (apps/redis) for caching, queues, locks, rate limiting.
• Polyglot shared libraries (Rust, Go) for specialized parsing (HTML→Markdown, PDF parsing, crawler logic).
• SDKs in JavaScript, Python, Rust for client integrations.
• A React UI (apps/ui/ingestion-ui) for manual ingestion; a static marketing/site (apps/www).
• Test-suite and CI/CD infra (GitHub Actions, Docker Compose, k8s manifests).
File structure
• Top-level “apps” folder groups each deployable component:
• api: main service
• playwright-service-ts: headless browser service
• redis: self-hosted Redis image + config
• js-sdk / python-sdk / rust-sdk: client libraries
• ui/ingestion-ui: React front end
• www: static website
• test-suite: integration/load tests
• SharedLibs under apps/api/sharedLibs are compiled Rust/Go modules invoked by the API for heavy-lifting.
• Dockerfiles, docker-compose.yaml, k8s yamls indicate container-based deployment and orchestration.
README for additional insights
• Confirms API endpoints (v0, v1 controllers), features (crawl, scrape, map, search, extract, batch, LLM extraction, actions).
• Lists external integrations: OpenAI/LLM providers, Google/SearxNG search, Stripe billing, Supabase ledger, Posthog analytics, Sentry, Slack notifications.