How kcharvi/firecrawl-with-ai-agents works
Project type and purpose
• This is a hybrid full-stack platform that bundles:
• A set of Python-based AI agents (in the ai_agents folder) for automating GitHub issue triage and labeling using large-language models.
• The Firecrawl “code_base” monorepo, which provides a TypeScript/Node.js web API for web crawling, scraping, extraction, queue processing, and billing.
• Polyglot client SDKs (JavaScript, Python, Rust) and a small React-based ingestion UI.
• Infrastructure definitions (Docker Compose, Kubernetes examples) and data-population/analysis scripts under repo_data.
Key directories & architectural patterns
• ai_agents/
• agents/issue_triage: Python modules that fetch GitHub issues, run prompts through an LLM chain, and then label or comment back.
• configs/: agent‐ and model‐level configuration.
• models/: adapters to different LLM backends (e.g., Gemini via LangChain).
• prompts/: prompt templates for triage tasks.
Pattern: Python package, uses Langchain, follows modular agent + prompt + model configuration.
• code_base/apps/api
• Monolithic HTTP JSON API implemented in Node.js/TypeScript.
• Exposes v0 & v1 REST endpoints for crawling, scraping, extraction, search, etc.
• Internally uses:
– Redis for job queues and caching
– Supabase for persistence
– Google Cloud Storage for archives
– Queue workers for batch/extraction jobs
– External LLM integration via generic-ai wrapper
– Billing & rate-limit services (Stripe, PostHog, custom billing queues)
Pattern: API + background workers + message queue + layered lib modules (controllers, services, lib/)
• code_base/apps/*-sdk & apps/ui
• SDKs (js, python, rust) wrap the REST API and handle auth/serialization.
• ingestion-ui: a React/Vite front end calling the API for demonstration.
Pattern: client libraries & demo UI in separate subprojects, each with its own build/test config.
• Infrastructure/config
• docker-compose.yaml orchestrates api, redis, supabase, etc.
• Kubernetes manifests under examples for cloud deploy.