# kcharvi/firecrawl-with-ai-agents architecture

> Architecture diagram of the GitHub repository kcharvi/firecrawl-with-ai-agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kcharvi/firecrawl-with-ai-agents
- Repository: https://github.com/kcharvi/firecrawl-with-ai-agents

## Overview

##### Project type and purpose
- This is a hybrid full-stack platform that bundles:
- A set of Python-based AI agents (in the ai_agents folder) for automating GitHub issue triage and labeling using large-language models.
- The Firecrawl “code_base” monorepo, which provides a TypeScript/Node.js web API for web crawling, scraping, extraction, queue processing, and billing.
- Polyglot client SDKs (JavaScript, Python, Rust) and a small React-based ingestion UI.
- Infrastructure definitions (Docker Compose, Kubernetes examples) and data-population/analysis scripts under repo_data.

##### Key directories & architectural patterns
- ai_agents/
- agents/issue_triage: Python modules that fetch GitHub issues, run prompts through an LLM chain, and then label or comment back.
- configs/: agent‐ and model‐level configuration.
- models/: adapters to different LLM backends (e.g., Gemini via LangChain).
- prompts/: prompt templates for triage tasks.
Pattern: Python package, uses Langchain, follows modular agent + prompt + model configuration.

- code_base/apps/api
- Monolithic HTTP JSON API implemented in Node.js/TypeScript.
- Exposes v0 & v1 REST endpoints for crawling, scraping, extraction, search, etc.
- Internally uses:
– Redis for job queues and caching
– Supabase for persistence
– Google Cloud Storage for archives
– Queue workers for batch/extraction jobs
– External LLM integration via generic-ai wrapper
– Billing & rate-limit services (Stripe, PostHog, custom billing queues)
Pattern: API + background workers + message queue + layered lib modules (controllers, services, lib/)

- code_base/apps/*-sdk & apps/ui
- SDKs (js, python, rust) wrap the REST API and handle auth/serialization.
- ingestion-ui: a React/Vite front end calling the API for demonstration.
Pattern: client libraries & demo UI in separate subprojects, each with its own build/test config.

- Infrastructure/config
- docker-compose.yaml orchestrates api, redis, supabase, etc.
- Kubernetes manifests under examples for cloud deploy.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% AI Agents Tier
    subgraph "AI Agents Tier"
        subgraph "ai_agents package"
            AI["ai_agents package"]:::python
            IssueTriage["Issue Triage Agents"]:::python
            Configs["Agent Configurations"]:::python
            Models["LLM Backend Adapters"]:::python
            Prompts["Prompt Templates"]:::python
        end
    end

    %% Firecrawl Backend Tier
    subgraph "Firecrawl Backend Tier"
        subgraph "code_base/apps/api"
            API["HTTP JSON API"]:::node
            ControllersV0["REST Controllers v0"]:::node
            ControllersV1["REST Controllers v1"]:::node
            Services["Business Services"]:::node
            Lib["Shared Libraries & Helpers"]:::node
            Workers["Queue Workers"]:::node
        end
    end

    %% Data & Queue Tier
    subgraph "Data & Queue Tier"
        Redis[(Redis)]:::data
        Supabase[(Supabase)]:::data
        GCS[(Google Cloud Storage)]:::data
    end

    %% External Services
    subgraph "External Services"
        GitHub[(GitHub API)]:::external
        LLM[(LLM Provider)]:::external
        PostHog[(PostHog)]:::external
        Stripe[(Stripe)]:::external
    end

    %% Client Tier
    subgraph "Client Tier"
        JS_SDK["JavaScript SDK"]:::client
        Py_SDK["Python SDK"]:::client
        Rust_SDK["Rust SDK"]:::client
        UI["Ingestion UI"]:::client
    end

    %% Infrastructure Tier
    subgraph "Infrastructure Tier"
        DockerComp["Docker Compose"]:::infra
        K8s["Kubernetes Examples"]:::infra
        RepoData["repo_data scripts"]:::infra
    end

    %% Relationships
    IssueTriage -->|fetch issues| GitHub
    IssueTriage -->|call LLM| LLM
    IssueTriage -->|apply labels| GitHub
    IssueTriage -->|call crawl API| API

    JS_SDK -->|HTTP requests| API
    Py_SDK -->|HTTP requests| API
    Rust_SDK -->|HTTP requests| API
    UI -->|HTTP requests| API

    API -->|enqueue job| Redis
    API -->|cache| Redis
    Workers -->|dequeue job| Redis
    Workers -->|store results| Supabase
    Workers -->|store archives| GCS
    API -->|call LLM| LLM
    API -->|billing events| Stripe
    API -->|telemetry| PostHog

    DockerComp -.-> API
    DockerComp -.-> Redis
    DockerComp -.-> Supabase
    DockerComp -.-> GCS
    K8s -.-> API
    K8s -.-> Redis
    K8s -.-> Supabase
    K8s -.-> GCS
    RepoData -->|analyze data| Supabase

    %% Click Events
    click AI "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/ai_agents/"
    click IssueTriage "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/ai_agents/agents/issue_triage/"
    click Configs "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/ai_agents/configs/"
    click Models "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/ai_agents/models/"
    click Prompts "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/ai_agents/prompts/"
    click API "https://github.com/kcharvi/firecrawl-with-ai-agents/blob/main/code_base/apps/api/src/index.ts"
    click ControllersV0 "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/api/src/controllers/v0/"
    click ControllersV1 "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/api/src/controllers/v1/"
    click Services "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/api/src/services/"
    click Lib "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/api/src/lib/"
    click Workers "https://github.com/kcharvi/firecrawl-with-ai-agents/blob/main/code_base/apps/api/src/services/queue-worker.ts"
    click JS_SDK "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/js-sdk/firecrawl/"
    click Py_SDK "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/python-sdk/firecrawl/"
    click Rust_SDK "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/rust-sdk/src/"
    click UI "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/apps/ui/ingestion-ui/"
    click DockerComp "https://github.com/kcharvi/firecrawl-with-ai-agents/blob/main/code_base/docker-compose.yaml"
    click K8s "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/code_base/examples/kubernetes/cluster-install/"
    click RepoData "https://github.com/kcharvi/firecrawl-with-ai-agents/tree/main/repo_data/"

    %% Styles
    classDef python fill:#D0E8F2,stroke:#000,stroke-width:1px;
    classDef node fill:#D6F5D6,stroke:#000,stroke-width:1px;
    classDef data fill:#FFF2B2,stroke:#000,stroke-width:1px;
    classDef external fill:#E0E0E0,stroke:#000,stroke-width:1px;
    classDef client fill:#FFD8B2,stroke:#000,stroke-width:1px;
    classDef infra fill:#F0F0F0,stroke:#000,stroke-width:1px;
```
