# tharuneshwar-s/firecrawl architecture

> Architecture diagram of the GitHub repository tharuneshwar-s/firecrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tharuneshwar-s/firecrawl
- Repository: https://github.com/tharuneshwar-s/firecrawl

## Overview

##### Project type and purpose
- Firecrawl is a mono-repo for a SaaS “web crawling + data extraction” platform. It provides:
- A Node/TypeScript REST API (apps/api) that handles crawl, scrape, map, search, extract endpoints.
- A headless-browser microservice (apps/playwright-service-ts) for JS-rendered sites.
- Redis (apps/redis) for caching, queues, locks, rate limiting.
- Polyglot shared libraries (Rust, Go) for specialized parsing (HTML→Markdown, PDF parsing, crawler logic).
- SDKs in JavaScript, Python, Rust for client integrations.
- A React UI (apps/ui/ingestion-ui) for manual ingestion; a static marketing/site (apps/www).
- Test-suite and CI/CD infra (GitHub Actions, Docker Compose, k8s manifests).

##### File structure
- Top-level “apps” folder groups each deployable component:
- api: main service
- playwright-service-ts: headless browser service
- redis: self-hosted Redis image + config
- js-sdk / python-sdk / rust-sdk: client libraries
- ui/ingestion-ui: React front end
- www: static website
- test-suite: integration/load tests
- SharedLibs under apps/api/sharedLibs are compiled Rust/Go modules invoked by the API for heavy-lifting.
- Dockerfiles, docker-compose.yaml, k8s yamls indicate container-based deployment and orchestration.

##### README for additional insights
- Confirms API endpoints (v0, v1 controllers), features (crawl, scrape, map, search, extract, batch, LLM extraction, actions).
- Lists external integrations: OpenAI/LLM providers, Google/SearxNG search, Stripe billing, Supabase ledger, Posthog analytics, Sentry, Slack notifications.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients
    subgraph "Clients"
        direction TB
        jsSDK["JS SDK"]:::frontend
        pySDK["Python SDK"]:::frontend
        rustSDK["Rust SDK"]:::frontend
        ui["Ingestion UI"]:::frontend
        www["Marketing Site"]:::frontend
    end

    %% Backend Services
    subgraph "Backend Services"
        direction TB
        subgraph "API Service (Node/TS)"
            direction TB
            API["API Service"]:::compute
            Dockerfile["Dockerfile"]:::compute
            Entrypoint["docker-entrypoint.sh"]:::compute
            ControllersV0["Controllers v0"]:::compute
            ControllersV1["Controllers v1"]:::compute
            Services["Services & Business Logic"]:::compute
            subgraph "SharedLibs" 
                direction TB
                crawler["Rust Crawler"]:::polyglot
                goMd["Go HTML→Markdown"]:::polyglot
                htmlTransformer["Rust HTML Transformer"]:::polyglot
                pdfParser["Rust PDF Parser"]:::polyglot
            end
        end

        playwright["Playwright Service"]:::compute
        Worker["Worker"]:::compute
    end

    %% Storage
    subgraph "Storage & Persistence"
        direction TB
        Redis["Redis Queue & Cache"]:::storage
        Supabase["Supabase (Postgres)"]:::storage
        GCS["Google Cloud Storage"]:::storage
    end

    %% External Services
    subgraph "External Dependencies"
        direction TB
        llm["LLM APIs"]:::external
        search["Search APIs"]:::external
        stripe["Stripe Billing"]:::external
        posthog["Posthog Analytics"]:::external
        sentry["Sentry"]:::external
        slack["Slack / Email / Webhook"]:::external
    end

    %% Client to API
    jsSDK -->|"HTTP(v1/v0)"| API
    pySDK -->|"HTTP(v1/v0)"| API
    rustSDK -->|"HTTP(v1/v0)"| API
    ui -->|"HTTP(v1/v0)"| API

    %% API internal structure
    API --> ControllersV0
    API --> ControllersV1
    API --> Services
    API --> Redis
    API --> playwright
    API --> crawler
    API --> goMd
    API --> htmlTransformer
    API --> pdfParser
    API --> Supabase
    API --> GCS
    API --> llm
    API --> search
    API --> stripe
    API --> posthog
    API --> sentry
    API --> slack

    %% Job flow
    API -->|"enqueue job"| Redis
    Redis -->|"dequeue job"| Worker
    Worker -->|"invoke headless"| playwright
    Worker -->|"use polyglot libs"| crawler
    Worker -->|"use polyglot libs"| goMd
    Worker -->|"use polyglot libs"| htmlTransformer
    Worker -->|"use polyglot libs"| pdfParser
    Worker -->|"store artifacts"| GCS
    Worker -->|"update status"| Redis

    %% API reads
    API -->|"fetch metadata"| Supabase
    API -->|"fetch results"| GCS

    %% Click Events
    click API "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api"
    click Dockerfile "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/Dockerfile"
    click Entrypoint "https://github.com/tharuneshwar-s/firecrawl/blob/main/apps/api/docker-entrypoint.sh"
    click ControllersV0 "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/src/controllers/v0"
    click ControllersV1 "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/src/controllers/v1"
    click Services "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/src/services"
    click crawler "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/sharedLibs/crawler"
    click goMd "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/sharedLibs/go-html-to-md"
    click htmlTransformer "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/sharedLibs/html-transformer"
    click pdfParser "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/api/sharedLibs/pdf-parser"
    click playwright "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/playwright-service-ts"
    click Redis "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/redis"
    click jsSDK "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/js-sdk/firecrawl"
    click pySDK "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/python-sdk/firecrawl"
    click rustSDK "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/rust-sdk"
    click ui "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/ui/ingestion-ui"
    click www "https://github.com/tharuneshwar-s/firecrawl/tree/main/apps/www"

    %% Styles
    classDef compute fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef storage fill:#DFFFE0,stroke:#28a745,color:#28a745
    classDef external fill:#FFF4E5,stroke:#d97504,color:#d97504
    classDef polyglot fill:#F3E0FF,stroke:#6f42c1,color:#6f42c1
    classDef frontend fill:#E0F7FF,stroke:#17a2b8,color:#17a2b8
```
