# eddygiusepe/firecrawl architecture

> Architecture diagram of the GitHub repository eddygiusepe/firecrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eddygiusepe/firecrawl
- Repository: https://github.com/eddygiusepe/firecrawl

## Overview

##### Project Type and Purpose
- Firecrawl is an API service whose purpose is to take a given URL, crawl the webpage, and convert its content into clean markdown.
- Although the file tree includes additional components (e.g., “chat_with_website.py” and a structured outputs module), the core functionality—as clearly stated in the README—is the firecrawl API that handles webpage processing.

##### Analyzing the File Structure
- At the top level, you have configuration and project files (e.g., .gitignore, pyproject.toml, LICENSE, uv.lock) which suggest a Python project possibly leveraging modern dependency and build systems. The presence of uv.lock hints at usage of components like Uvicorn or related tooling.
- The folder “1_Build_a_Chat_with_website_using_Groq_Llama3” contains a script for building a chat component using website data. This might be an additional (or extended) feature not central to the Firecrawl API.
- Similarly, “2_Structured_Outputs_and_JSON_Strict_Mode” includes a script for handling structured outputs in JSON possibly as part of the API’s data processing or response formatting workflow.
- The “1_firecrawl.ipynb” notebook is likely used for demonstration or experiment purposes with the Firecrawl functionality.

##### Key Components
a. API Service Layer (Firecrawl API):
– Central service which receives URL requests via an HTTP endpoint.
– It handles routing, request validation, and response delivery.
– Mark this as the “Front Door” of the system.

b. Web Crawler/Fetcher Module:
– A component that takes the provided URL and performs the retrieval of webpage content.
– This module is responsible for handling HTTP requests to external sites, error management, etc.

c. Content Processing & Markdown Conversion Module:
– After fetching the raw content, this component processes HTML (or possibly other formats) to clean and extract text.
– It then converts this content into markdown format.
– You might illustrate any specific libraries or transformation pipelines (for example, if any third-party parser is used).

d. Output Formatter / Structured Output Module:
– Inference from the “Structured_Outputs_and_JSON_Strict_Mode” directory is that there is a utility to ensure outputs (or even responses to API calls) are structured, possibly as JSON.
– This module may ensure that both the markdown results and any additional metadata are formatted according to expected strict schemas.

e. (Optional) Chat Integration & Extended Functionality:
– The “Build_a_Chat_with_website_using_Groq_Llama3” directory signals that there is an auxiliary feature to integrate chat functionalities which likely builds on the webpage content.
– While not core to the firecrawl API, it can be depicted as an extension or a separate service that takes markdown or website insights from Firecrawl for further natural language processing or user interactions.

##### Establishing Relationships and Data Flow
1. Client Request enters the API endpoint.
2. The API layer routes the URL to the Web Crawler module.
3. The Web Crawler fetches raw HTML data.
4. The data moves to the Content Processing module, where conversion to markdown occurs.
5. Optionally, the processed data is also formatted using the Structured Output module.
6. The final clean markdown (and any formatted data) is sent back as the API response.

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client("Client Request"):::external
    API("API Service (Front Door)"):::api
    WC("Web Crawler/Fetcher Module"):::module
    CP("Content Processing & Markdown Conversion"):::module
    OF("Output Formatter / Structured Output"):::module
    Chat("Chat Integration & Extended Functionality"):::optional

    Client -->|"sendsURL"| API
    API -->|"forwardsURL"| WC
    WC -->|"fetchesHTML"| CP
    CP -->|"convertsToMarkdown"| OF
    OF -->|"formatsResponse"| API
    CP -->|"sendsContent"| Chat

    click API "https://github.com/eddygiusepe/firecrawl/blob/main/1_firecrawl.ipynb"
    click WC "https://github.com/eddygiusepe/firecrawl/blob/main/1_firecrawl.ipynb"
    click CP "https://github.com/eddygiusepe/firecrawl/blob/main/1_firecrawl.ipynb"
    click OF "https://github.com/eddygiusepe/firecrawl/blob/main/2_Structured_Outputs_and_JSON_Strict_Mode/OpenAI_Structured_Outputs_and_JSON_Strict_Mode.py"
    click Chat "https://github.com/eddygiusepe/firecrawl/blob/main/1_Build_a_Chat_with_website_using_Groq_Llama3/chat_with_website.py"

    classDef api fill:#c1d2f0,stroke:#000,stroke-width:2px
    classDef module fill:#e8f4dc,stroke:#305c22,stroke-width:2px
    classDef optional fill:#ffdfba,stroke:#d35400,stroke-width:2px
    classDef external fill:#fdebd0,stroke:#c0392b,stroke-width:2px
```
