# mendableai/firecrawl architecture

> Architecture diagram of the GitHub repository mendableai/firecrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mendableai/firecrawl
- Repository: https://github.com/mendableai/firecrawl

## Overview

##### Main System Components

a) Client Layer:
- Multiple SDK implementations (Python, Node.js, Rust, Go)
- REST API endpoints
- UI components (ingestion-ui)

b) API Service Layer:
- API Gateway (/v0 and /v1 endpoints)
- Authentication/Authorization
- Rate Limiting
- Request Validation

c) Core Services:
- Scraping Service
- Crawling Service
- Mapping Service
- Search Service
- LLM Extraction Service
- Queue Management

d) Infrastructure Components:
- Redis (for caching and job management)
- Playwright Service (for JavaScript rendering)
- Database (Supabase)
- Job Queue System

##### Component Relationships

a) Data Flow:
- Client requests → API Gateway → Core Services
- Core Services → Infrastructure Components
- Results → Client Response

b) Service Integration:
- Queue Worker → Scraping/Crawling Services
- Playwright Service ↔ Scraping Service
- Redis Cache ↔ All Services
- Database ↔ All Services

##### Specific Implementation Details

Specialized components:

a) Scraping Engines:
- Cache Engine
- Fetch Engine
- Playwright Engine
- PDF Engine
- DOCX Engine
- Fire Engine
- ScrapingBee Engine

b) Processing Pipeline:
- URL Validation
- Content Extraction
- HTML to Markdown Conversion
- Metadata Extraction
- LLM Processing
- Result Formatting

##### Architectural Patterns

Highlight:

- Microservices architecture
- Queue-based job processing
- Caching strategies
- Rate limiting implementation
- Error handling flow
- Monitoring points

##### Error handling and recovery paths

This will create a comprehensive visualization that accurately represents Firecrawl's architecture while maintaining clarity and usefulness for technical discussions and system understanding.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client
        PythonSDK["Python SDK"]:::client
        NodeSDK["Node.js SDK"]:::client
        RustSDK["Rust SDK"]:::client
        GoSDK["Go SDK"]:::client
        UI["Ingestion UI"]:::client
    end

    subgraph APIGateway
        V0["V0 API"]:::api
        V1["V1 API"]:::api
        Auth["Authentication"]:::security
        RateLimit["Rate Limiter"]:::security
        Validation["Request Validation"]:::api
    end

    subgraph CoreServices
        Scraping["Scraping Service"]:::core
        Crawling["Crawling Service"]:::core
        Mapping["Mapping Service"]:::core
        Search["Search Service"]:::core
        LLM["LLM Extraction"]:::core
        Queue["Queue Management"]:::core
    end

    subgraph Infrastructure
        Redis["Redis Cache"]:::infra
        Playwright["Playwright Service"]:::infra
        DB["Supabase DB"]:::infra
        JobQueue["Job Queue"]:::infra
    end

    subgraph ScrapingEngines
        Cache["Cache Engine"]:::engine
        Fetch["Fetch Engine"]:::engine
        PlaywrightE["Playwright Engine"]:::engine
        PDF["PDF Engine"]:::engine
        DOCX["DOCX Engine"]:::engine
        Fire["Fire Engine"]:::engine
        ScrapingBee["ScrapingBee Engine"]:::engine
    end

    %% Client connections
    PythonSDK --> V0
    NodeSDK --> V0
    RustSDK --> V0
    GoSDK --> V0
    UI --> V1

    %% API Gateway flow
    V0 --> Auth
    V1 --> Auth
    Auth --> RateLimit
    RateLimit --> Validation
    Validation --> CoreServices

    %% Core Services interactions
    Scraping --> ScrapingEngines
    Crawling --> ScrapingEngines
    Queue --> JobQueue
    Search --> DB
    LLM --> DB

    %% Infrastructure connections
    ScrapingEngines --> Redis
    ScrapingEngines --> Playwright
    CoreServices --> DB
    JobQueue --> Redis

    %% Styling
    classDef client fill:#a8d5ff
    classDef api fill:#95ff95
    classDef core fill:#ffb366
    classDef infra fill:#ff9999
    classDef engine fill:#dfff80
    classDef security fill:#ff80b3

    %% Click events
    click PythonSDK "https://github.com/mendableai/firecrawl/tree/main/apps/python-sdk/"
    click UI "https://github.com/mendableai/firecrawl/tree/main/apps/ui/ingestion-ui/"
    click V0 "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/routes/v0.ts"
    click V1 "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/routes/v1.ts"
    click Auth "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/controllers/auth.ts"
    click RateLimit "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/services/rate-limiter.ts"
    click Scraping "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/"
    click Crawling "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/WebScraper/"
    click Mapping "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/controllers/v1/map.ts"
    click Search "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/search/"
    click LLM "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/lib/LLM-extraction/"
    click Queue "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/services/queue-service.ts"
    click Redis "https://github.com/mendableai/firecrawl/tree/main/apps/redis/"
    click Playwright "https://github.com/mendableai/firecrawl/tree/main/apps/playwright-service/"
    click DB "https://github.com/mendableai/firecrawl/blob/main/apps/api/src/services/supabase.ts"
    click Cache "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/cache/"
    click Fetch "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/fetch/"
    click PlaywrightE "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/playwright/"
    click PDF "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/pdf/"
    click DOCX "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/docx/"
    click Fire "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/fire-engine/"
    click ScrapingBee "https://github.com/mendableai/firecrawl/tree/main/apps/api/src/scraper/scrapeURL/engines/scrapingbee/"
```
