# gramosoftai/gcrawlai architecture

> Architecture diagram of the GitHub repository gramosoftai/gcrawlai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gramosoftai/gcrawlai
- Repository: https://github.com/gramosoftai/gcrawlai

## Overview

GcrawlAI is a web crawling, scraping, and search platform. A client submits scrape, crawl, link-discovery, or search requests to the API; the API validates access and limits, then runs or queues work. Crawling uses browser and page-processing components, with separate mapping and search capabilities; results and progress use database, object-storage, or Redis-backed paths. The README documents additional extraction and stealth features whose full implementation was not sampled, so those links are shown only where documented or supported by the source index.

Main entry points: [`api/routes/gcrawl_routes/crawler_routes.py`](https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/crawler_routes.py) (Crawl endpoints), [`api/core/database.py`](https://github.com/gramosoftai/gcrawlai/blob/main/api/core/database.py) (PostgreSQL data), [`web_crawler/crawler/helpers/crawler_offline.py`](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/crawler_offline.py) (Offline bundles), [`web_crawler/crawler/page/page_crawler1.py`](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/page/page_crawler1.py) (Page extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and accounts"]
  node_app["API application<br/>[api.py]"]
  node_crawl_api["Crawl endpoints<br/>[crawler_routes.py]"]
  node_search_api["Search endpoint<br/>[search_routes.py]"]
  node_auth_api["Auth endpoints<br/>[auth_routes.py]"]
  node_cookies["Cookie sync<br/>[cookie_routes.py]"]
  node_keys["API key management<br/>[api_key_routes.py]"]
  node_payloads["Request models<br/>[gcrawl_payloads.py]"]
  node_security["Access and plan checks<br/>[security.py]"]
  node_auth["Account authentication<br/>[auth_manager.py]"]
end

subgraph group_jobs["Crawl execution"]
  node_queue["Priority task queue<br/>[queue_manager.py]"]
  node_tasks["Celery task workers<br/>[celery_tasks.py]"]
  node_coordinator["Crawl coordinator<br/>[crawler.py]"]
  node_browser_crawler["Browser crawler<br/>[web_crawler.py]"]
end

subgraph group_extract["Extraction and discovery"]
  node_page["Page extraction<br/>[page_crawler1.py]"]
  node_markdown["Content processing"]
  node_screenshots["Screenshot capture"]
  node_offline["Offline bundles<br/>[crawler_offline.py]"]
  node_mapping["URL mapping<br/>[map_crawler.py]"]
  node_search["Search routing<br/>[search_engine.py]"]
  node_google["Google search<br/>[google_search.py]"]
end

subgraph group_platform["Shared services"]
  node_proxy["Proxy rotation<br/>[proxy_manager.py]"]
  node_database[("PostgreSQL data<br/>[database.py]")]
  node_events["Redis crawl events<br/>[redis_events.py]"]
  node_email["Email notifications<br/>[email_service.py]"]
  node_config["Crawl configuration<br/>[config.py]"]
end

node_client(("API client"))

node_client -->|"calls API"| node_app
node_app -->|"routes requests"| node_crawl_api
node_app -->|"routes requests"| node_search_api
node_app -->|"routes requests"| node_auth_api
node_crawl_api -->|"validates payloads"| node_payloads
node_crawl_api -->|"checks access"| node_security
node_crawl_api -->|"queues work"| node_queue
node_crawl_api -->|"dispatches crawl"| node_tasks
node_crawl_api -->|"runs bundles"| node_offline
node_crawl_api -->|"discovers links"| node_mapping
node_tasks -->|"runs crawl"| node_coordinator
node_coordinator -->|"delegates crawling"| node_browser_crawler
node_tasks -->|"publishes progress"| node_events
node_coordinator -->|"publishes events"| node_events
node_tasks -->|"updates job data"| node_database
node_browser_crawler -.->|"extracts pages"| node_page
node_page -.->|"cleans content"| node_markdown
node_page -.->|"captures screenshots"| node_screenshots
node_crawl_api -->|"scrapes pages"| node_page
node_crawl_api -->|"configures crawling"| node_config
node_search_api -->|"runs search"| node_search
node_search -->|"dispatches search"| node_google
node_google -->|"routes via proxy"| node_proxy
node_mapping -->|"uses proxy"| node_proxy
node_offline -->|"uses page crawler"| node_page
node_offline -->|"uses proxy"| node_proxy
node_offline -->|"publishes progress"| node_events
node_offline -->|"stores crawl data"| node_database
node_keys -->|"shares key checks"| node_security
node_keys -->|"manages keys"| node_database
node_cookies -->|"stores session data"| node_database
node_auth_api -->|"validates payloads"| node_payloads
node_auth_api -->|"checks access"| node_security
node_auth -->|"sends account mail"| node_email
node_auth -->|"reads account data"| node_database
node_proxy -->|"reads ISP data"| node_database

click node_app "https://github.com/gramosoftai/gcrawlai/blob/main/api/api.py"
click node_crawl_api "https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/crawler_routes.py"
click node_search_api "https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/search_routes.py"
click node_auth_api "https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/auth_routes.py"
click node_cookies "https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/cookie_routes.py"
click node_keys "https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/api_key_routes.py"
click node_payloads "https://github.com/gramosoftai/gcrawlai/blob/main/api/models/gcrawl_payloads.py"
click node_security "https://github.com/gramosoftai/gcrawlai/blob/main/api/core/security.py"
click node_auth "https://github.com/gramosoftai/gcrawlai/blob/main/api/auth/auth_manager.py"
click node_queue "https://github.com/gramosoftai/gcrawlai/blob/main/api/services/queue_manager.py"
click node_tasks "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/celery_tasks.py"
click node_coordinator "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/crawler.py"
click node_browser_crawler "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/web_crawler.py"
click node_page "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/page/page_crawler1.py"
click node_markdown "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/content_processor.py"
click node_screenshots "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/crawler_screenshot.py"
click node_offline "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/crawler_offline.py"
click node_mapping "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/map/map_crawler.py"
click node_search "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/search/search_engine.py"
click node_google "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/search/google_search.py"
click node_proxy "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/common/proxy_manager.py"
click node_database "https://github.com/gramosoftai/gcrawlai/blob/main/api/core/database.py"
click node_events "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/common/redis_events.py"
click node_email "https://github.com/gramosoftai/gcrawlai/blob/main/api/services/email_service.py"
click node_config "https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/common/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_crawl_api,node_search_api,node_auth_api,node_cookies,node_keys,node_payloads,node_security,node_auth,node_client toneBlue
class node_queue,node_tasks,node_coordinator,node_browser_crawler toneAmber
class node_page,node_markdown,node_screenshots,node_offline,node_mapping,node_search,node_google toneMint
class node_proxy,node_database,node_events,node_email,node_config toneRose
```

## Components

### API and accounts

- [API application](https://github.com/gramosoftai/gcrawlai/blob/main/api/api.py): `api/api.py`
- [Crawl endpoints](https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/crawler_routes.py): `api/routes/gcrawl_routes/crawler_routes.py`
- [Search endpoint](https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/search_routes.py): `api/routes/gcrawl_routes/search_routes.py`
- [Auth endpoints](https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/auth_routes.py): `api/routes/gcrawl_routes/auth_routes.py`
- [Cookie sync](https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/cookie_routes.py): `api/routes/gcrawl_routes/cookie_routes.py`
- [API key management](https://github.com/gramosoftai/gcrawlai/blob/main/api/routes/gcrawl_routes/api_key_routes.py): `api/routes/gcrawl_routes/api_key_routes.py`
- [Request models](https://github.com/gramosoftai/gcrawlai/blob/main/api/models/gcrawl_payloads.py): `api/models/gcrawl_payloads.py`
- [Access and plan checks](https://github.com/gramosoftai/gcrawlai/blob/main/api/core/security.py): `api/core/security.py`
- [Account authentication](https://github.com/gramosoftai/gcrawlai/blob/main/api/auth/auth_manager.py): `api/auth/auth_manager.py`

### Crawl execution

- [Priority task queue](https://github.com/gramosoftai/gcrawlai/blob/main/api/services/queue_manager.py): `api/services/queue_manager.py`
- [Celery task workers](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/celery_tasks.py): `web_crawler/crawler/celery_tasks.py`
- [Crawl coordinator](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/crawler.py): `web_crawler/crawler/crawler.py`
- [Browser crawler](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/web_crawler.py): `web_crawler/crawler/web_crawler.py`

### Extraction and discovery

- [Page extraction](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/page/page_crawler1.py): `web_crawler/crawler/page/page_crawler1.py`
- [Content processing](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/content_processor.py): `web_crawler/crawler/helpers/content_processor.py`
- [Screenshot capture](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/crawler_screenshot.py): `web_crawler/crawler/helpers/crawler_screenshot.py`
- [Offline bundles](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/helpers/crawler_offline.py): `web_crawler/crawler/helpers/crawler_offline.py`
- [URL mapping](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/crawler/map/map_crawler.py): `web_crawler/crawler/map/map_crawler.py`
- [Search routing](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/search/search_engine.py): `web_crawler/search/search_engine.py`
- [Google search](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/search/google_search.py): `web_crawler/search/google_search.py`

### Shared services

- [Proxy rotation](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/common/proxy_manager.py): `web_crawler/common/proxy_manager.py`
- [PostgreSQL data](https://github.com/gramosoftai/gcrawlai/blob/main/api/core/database.py): `api/core/database.py`
- [Redis crawl events](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/common/redis_events.py): `web_crawler/common/redis_events.py`
- [Email notifications](https://github.com/gramosoftai/gcrawlai/blob/main/api/services/email_service.py): `api/services/email_service.py`
- [Crawl configuration](https://github.com/gramosoftai/gcrawlai/blob/main/web_crawler/common/config.py): `web_crawler/common/config.py`

### Other components

- **API client**

## Connections

- API client → API application: calls API
- API application → Crawl endpoints: routes requests (evidence: `api/api.py`)
- API application → Search endpoint: routes requests (evidence: `api/api.py`)
- API application → Auth endpoints: routes requests (evidence: `api/api.py`)
- Crawl endpoints → Request models: validates payloads (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Crawl endpoints → Access and plan checks: checks access (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Crawl endpoints → Priority task queue: queues work (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Crawl endpoints → Celery task workers: dispatches crawl (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Crawl endpoints → Offline bundles: runs bundles (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Crawl endpoints → URL mapping: discovers links
- Celery task workers → Crawl coordinator: runs crawl (evidence: `web_crawler/crawler/celery_tasks.py`)
- Crawl coordinator → Browser crawler: delegates crawling (evidence: `web_crawler/crawler/crawler.py`)
- Celery task workers → Redis crawl events: publishes progress (evidence: `web_crawler/crawler/celery_tasks.py`)
- Crawl coordinator → Redis crawl events: publishes events (evidence: `web_crawler/crawler/crawler.py`)
- Celery task workers → PostgreSQL data: updates job data (evidence: `web_crawler/crawler/celery_tasks.py`)
- Browser crawler → Page extraction: extracts pages
- Page extraction → Content processing: cleans content
- Page extraction → Screenshot capture: captures screenshots
- Crawl endpoints → Page extraction: scrapes pages (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Crawl endpoints → Crawl configuration: configures crawling (evidence: `api/routes/gcrawl_routes/crawler_routes.py`)
- Search endpoint → Search routing: runs search (evidence: `api/routes/gcrawl_routes/search_routes.py`)
- Search routing → Google search: dispatches search (evidence: `web_crawler/search/search_engine.py`)
- Google search → Proxy rotation: routes via proxy (evidence: `web_crawler/search/google_search.py`)
- URL mapping → Proxy rotation: uses proxy (evidence: `web_crawler/crawler/map/map_crawler.py`)
- Offline bundles → Page extraction: uses page crawler (evidence: `web_crawler/crawler/helpers/crawler_offline.py`)
- Offline bundles → Proxy rotation: uses proxy (evidence: `web_crawler/crawler/helpers/crawler_offline.py`)
- Offline bundles → Redis crawl events: publishes progress (evidence: `web_crawler/crawler/helpers/crawler_offline.py`)
- Offline bundles → PostgreSQL data: stores crawl data (evidence: `web_crawler/crawler/helpers/crawler_offline.py`)
- API key management → Access and plan checks: shares key checks (evidence: `api/routes/gcrawl_routes/api_key_routes.py`)
- API key management → PostgreSQL data: manages keys (evidence: `api/routes/gcrawl_routes/api_key_routes.py`)
- Cookie sync → PostgreSQL data: stores session data (evidence: `api/routes/gcrawl_routes/cookie_routes.py`)
- Auth endpoints → Request models: validates payloads (evidence: `api/routes/gcrawl_routes/auth_routes.py`)
- Auth endpoints → Access and plan checks: checks access (evidence: `api/routes/gcrawl_routes/auth_routes.py`)
- Account authentication → Email notifications: sends account mail (evidence: `api/auth/auth_manager.py`)
- Account authentication → PostgreSQL data: reads account data (evidence: `api/auth/auth_manager.py`)
- Proxy rotation → PostgreSQL data: reads ISP data (evidence: `web_crawler/common/proxy_manager.py`)
