How mendableai/firecrawl works
Main System Components
a) Client Layer:
• Multiple SDK implementations (Python, Node.js, Rust, Go)
• REST API endpoints
• UI components (ingestion-ui)
b) API Service Layer:
• API Gateway (/v0 and /v1 endpoints)
• Authentication/Authorization
• Rate Limiting
• Request Validation
c) Core Services:
• Scraping Service
• Crawling Service
• Mapping Service
• Search Service
• LLM Extraction Service
• Queue Management
d) Infrastructure Components:
• Redis (for caching and job management)
• Playwright Service (for JavaScript rendering)
• Database (Supabase)
• Job Queue System
Component Relationships
a) Data Flow:
• Client requests → API Gateway → Core Services
• Core Services → Infrastructure Components
• Results → Client Response
b) Service Integration:
• Queue Worker → Scraping/Crawling Services
• Playwright Service ↔ Scraping Service
• Redis Cache ↔ All Services
• Database ↔ All Services
Specific Implementation Details
Specialized components:
a) Scraping Engines:
• Cache Engine
• Fetch Engine
• Playwright Engine
• PDF Engine
• DOCX Engine
• Fire Engine
• ScrapingBee Engine
b) Processing Pipeline:
• URL Validation
• Content Extraction
• HTML to Markdown Conversion
• Metadata Extraction
• LLM Processing
• Result Formatting
Architectural Patterns
Highlight:
• Microservices architecture
• Queue-based job processing
• Caching strategies
• Rate limiting implementation
• Error handling flow
• Monitoring points
Error handling and recovery paths
This will create a comprehensive visualization that accurately represents Firecrawl's architecture while maintaining clarity and usefulness for technical discussions and system understanding.