# bazinga-web/firecrawl architecture

> Architecture diagram of the GitHub repository bazinga-web/firecrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bazinga-web/firecrawl
- Repository: https://github.com/bazinga-web/firecrawl

## Overview

##### Project Type and Overall Purpose
- This is a full‐stack, API-first web scraping and crawling platform with multiple SDKs and a self-hostable codebase.
- Its core functionality is to crawl websites, scrape web content, extract structured data (including LLM extraction), and support various actions (e.g., interacting with dynamic content) before delivering clean, LLM-ready outputs.
- The monorepo contains backend services (the API), multiple language SDKs (JavaScript, Python, Rust, and even Go libraries within shared libs), a UI ingestion app, containerized services (like Redis), and even a Playwright service.

##### Main Components (from the file tree and README)
- API Service: Located in “apps/api” – the heart of the system with controllers (v0, v1), routes, libraries for crawling/scraping/LLM extraction, helper modules, third-party integrations (billing, notifications, etc.), and a Dockerfile for containerization.
- SDKs: Client libraries in “apps/js-sdk”, “apps/python-sdk”, “apps/rust-sdk” that wrap API endpoints enabling developers to integrate Firecrawl anywhere.
- UI/Frontend: The Ingestion UI located in “apps/ui/ingestion-ui”, which likely provides a user-friendly interface for testing, managing scrape/crawl jobs or visualizing results.
- Additional Services:
– Playwright Service (“apps/playwright-service-ts”) for handling advanced browser automation tasks.
– Redis service (“apps/redis”) used for caching or queue management.
– An internal test suite (“apps/test-suite”) that ensures the system components work together.
- External Integrations:
– LLM extraction components (embedded in scraping and extract endpoints) that may interface with external LLM services or use custom in-house logic for transformation.
– Third-party API requests, along with integrations into various frameworks (e.g., Langchain, Crew.ai, etc.) as demonstrated in the documentation.

##### Relationships and Interactions
- Clients (either via SDKs, UI, or API calls directly) interact with the API Service.
- The API service receives requests for scraping, crawling, mapping, and extraction. It then delegates these tasks to specialized internal modules:
– Scrape Controller: Orchestrates fetching the website’s content; interacts with scraping libraries and possibly the Playwright service for dynamic content.
– Crawl Controller: Handles recursive job submission to traverse subpages.
– Extract / LLM Extraction Module: Processes HTML into markdown or JSON structures, optionally leveraging LLM-based processing.
- API Service communicates with a Redis instance (and possibly other message queue services) to manage concurrency, caching, and job status.
- Results are returned via APIs, with the SDKs providing language-specific wrappers.
- Other components (billing, logging, notification services) integrate within the API to support ancillary operations.

##### Architectural Patterns and Design Principles
- Modular design: The file structure shows clearly separable modules and libraries (e.g., “controllers”, “lib”, “services”, “scraper”), hinting at a layered or service-oriented approach.
- Containerization & Deployment: Dockerfiles in multiple apps indicate that services are containerized—so include Docker containers or Kubernetes pods for API, Redis, and Playwright services.
- Extensibility: The presence of multiple officially supported SDKs (JS, Python, Rust) and external integrations imply a plug-and-play design that allows third-party applications to integrate via clearly defined API boundaries.
- Emphasis on scalability and asynchronous processing: Use of batch APIs and job IDs for crawl/scrape processes and the explicit use of Redis for rate limiting and job queues.

##### Final Touches
- Optionally, denote the use of asynchronous processing – for example, using a “job queue” label between API service and Redis.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        JS["JavaScript SDK"]:::client
        PY["Python SDK"]:::client
        RUST["Rust SDK"]:::client
        UI["UI Ingestion App"]:::client
    end

    %% API Service Layer with internal modules
    subgraph "API Service"
        API["API Service"]:::api
        SCRAPE["Scrape Controller<br/>(Fetches and processes HTML)"]:::apiModule
        CRAWL["Crawl Controller<br/>(Handles recursive job submission)"]:::apiModule
        EXTRACT["Extraction Module<br/>(Transforms HTML via LLM)"]:::apiModule
        API --- SCRAPE
        API --- CRAWL
        API --- EXTRACT
    end

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        REDIS["Redis Service<br/>(Job Queue & Caching)"]:::infra
        PLAY["Playwright Service<br/>(Dynamic content processing)"]:::infra
    end

    %% External Integrations and Testing
    EXTERNAL["External Integrations<br/>(LLM, Third-party APIs)"]:::external
    TEST["Internal Test Suite"]:::test

    %% Client requests to API Service
    JS -->|"API request"| API
    PY -->|"API request"| API
    RUST -->|"API request"| API
    UI -->|"UI actions"| API

    %% API Service internal communications and external integrations
    API -->|"communicates"| REDIS
    API -->|"delegates dynamic content"| PLAY
    API -->|"integrates"| EXTERNAL
    API -->|"triggers tests"| TEST

    %% Additional internal interactions within API modules
    SCRAPE -->|"fetches dynamic data"| PLAY

    %% Click Events
    click API "https://github.com/bazinga-web/firecrawl/tree/main/apps/api"
    click JS "https://github.com/bazinga-web/firecrawl/tree/main/apps/js-sdk"
    click PY "https://github.com/bazinga-web/firecrawl/tree/main/apps/python-sdk"
    click RUST "https://github.com/bazinga-web/firecrawl/tree/main/apps/rust-sdk"
    click UI "https://github.com/bazinga-web/firecrawl/tree/main/apps/ui/ingestion-ui"
    click PLAY "https://github.com/bazinga-web/firecrawl/tree/main/apps/playwright-service-ts"
    click REDIS "https://github.com/bazinga-web/firecrawl/tree/main/apps/redis"
    click TEST "https://github.com/bazinga-web/firecrawl/tree/main/apps/test-suite"

    %% Styles
    classDef client fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef api fill:#F9E79F,stroke:#AF601A,stroke-width:2px;
    classDef apiModule fill:#FAD7A0,stroke:#B9770E,stroke-width:2px;
    classDef infra fill:#A9DFBF,stroke:#117864,stroke-width:2px;
    classDef external fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef test fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
```
