How gramosoftai/gcrawlai works
GcrawlAI is a web crawling, scraping, and search platform. A client submits scrape, crawl, link-discovery, or search requests to the API; the API validates access and limits, then runs or queues work. Crawling uses browser and page-processing components, with separate mapping and search capabilities; results and progress use database, object-storage, or Redis-backed paths. The README documents additional extraction and stealth features whose full implementation was not sampled, so those links are shown only where documented or supported by the source index.
Main entry points: crawler_routes.py Crawl endpoints · database.py PostgreSQL data · crawler_offline.py Offline bundles · page_crawler1.py Page extraction
Components 26
API and accounts
- API application
api/api.py - Crawl endpoints
api/routes/gcrawl_routes/crawler_routes.py - Search endpoint
api/routes/gcrawl_routes/search_routes.py - Auth endpoints
api/routes/gcrawl_routes/auth_routes.py - Cookie sync
api/routes/gcrawl_routes/cookie_routes.py - API key management
api/routes/gcrawl_routes/api_key_routes.py - Request models
api/models/gcrawl_payloads.py - Access and plan checks
api/core/security.py - Account authentication
api/auth/auth_manager.py
Crawl execution
- Priority task queue
api/services/queue_manager.py - Celery task workers
web_crawler/crawler/celery_tasks.py - Crawl coordinator
web_crawler/crawler/crawler.py - Browser crawler
web_crawler/crawler/web_crawler.py
Extraction and discovery
- Page extraction
web_crawler/crawler/page/page_crawler1.py - Content processing
web_crawler/crawler/helpers/content_processor.py - Screenshot capture
web_crawler/crawler/helpers/crawler_screenshot.py - Offline bundles
web_crawler/crawler/helpers/crawler_offline.py - URL mapping
web_crawler/crawler/map/map_crawler.py - Search routing
web_crawler/search/search_engine.py - Google search
web_crawler/search/google_search.py
Shared services
- Proxy rotation
web_crawler/common/proxy_manager.py - PostgreSQL data
api/core/database.py - Redis crawl events
web_crawler/common/redis_events.py - Email notifications
api/services/email_service.py - Crawl configuration
web_crawler/common/config.py
Other components
- API client
Connections 36
- API client → API application: calls API
- API application → Crawl endpoints: routes requests
api/api.py - API application → Search endpoint: routes requests
api/api.py - API application → Auth endpoints: routes requests
api/api.py - Crawl endpoints → Request models: validates payloads
api/routes/gcrawl_routes/crawler_routes.py - Crawl endpoints → Access and plan checks: checks access
api/routes/gcrawl_routes/crawler_routes.py - Crawl endpoints → Priority task queue: queues work
api/routes/gcrawl_routes/crawler_routes.py - Crawl endpoints → Celery task workers: dispatches crawl
api/routes/gcrawl_routes/crawler_routes.py - Crawl endpoints → Offline bundles: runs bundles
api/routes/gcrawl_routes/crawler_routes.py - Crawl endpoints → URL mapping: discovers links
- Celery task workers → Crawl coordinator: runs crawl
web_crawler/crawler/celery_tasks.py - Crawl coordinator → Browser crawler: delegates crawling
web_crawler/crawler/crawler.py - Celery task workers → Redis crawl events: publishes progress
web_crawler/crawler/celery_tasks.py - Crawl coordinator → Redis crawl events: publishes events
web_crawler/crawler/crawler.py - Celery task workers → PostgreSQL data: updates job data
web_crawler/crawler/celery_tasks.py - Browser crawler → Page extraction: extracts pages
- Page extraction → Content processing: cleans content
- Page extraction → Screenshot capture: captures screenshots
- Crawl endpoints → Page extraction: scrapes pages
api/routes/gcrawl_routes/crawler_routes.py - Crawl endpoints → Crawl configuration: configures crawling
api/routes/gcrawl_routes/crawler_routes.py - Search endpoint → Search routing: runs search
api/routes/gcrawl_routes/search_routes.py - Search routing → Google search: dispatches search
web_crawler/search/search_engine.py - Google search → Proxy rotation: routes via proxy
web_crawler/search/google_search.py - URL mapping → Proxy rotation: uses proxy
web_crawler/crawler/map/map_crawler.py - Offline bundles → Page extraction: uses page crawler
web_crawler/crawler/helpers/crawler_offline.py - Offline bundles → Proxy rotation: uses proxy
web_crawler/crawler/helpers/crawler_offline.py - Offline bundles → Redis crawl events: publishes progress
web_crawler/crawler/helpers/crawler_offline.py - Offline bundles → PostgreSQL data: stores crawl data
web_crawler/crawler/helpers/crawler_offline.py - API key management → Access and plan checks: shares key checks
api/routes/gcrawl_routes/api_key_routes.py - API key management → PostgreSQL data: manages keys
api/routes/gcrawl_routes/api_key_routes.py - Cookie sync → PostgreSQL data: stores session data
api/routes/gcrawl_routes/cookie_routes.py - Auth endpoints → Request models: validates payloads
api/routes/gcrawl_routes/auth_routes.py - Auth endpoints → Access and plan checks: checks access
api/routes/gcrawl_routes/auth_routes.py - Account authentication → Email notifications: sends account mail
api/auth/auth_manager.py - Account authentication → PostgreSQL data: reads account data
api/auth/auth_manager.py - Proxy rotation → PostgreSQL data: reads ISP data
web_crawler/common/proxy_manager.py