# omkarcloud/website-email-contact-scraper architecture

> Architecture diagram of the GitHub repository omkarcloud/website-email-contact-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/omkarcloud/website-email-contact-scraper
- Repository: https://github.com/omkarcloud/website-email-contact-scraper

## Overview

This repository provides a website-contact scraper with both a Python callable/CLI and a local browser dashboard. The main workflow accepts website URLs and a crawl mode, crawls pages, extracts contact details and technologies, aggregates structured results, and presents task results for filtering and export. README-listed extraction capabilities are represented, but many internals are unsampled; edges are limited to relationships supported by the excerpts, SOURCE INDEX references, or README. The hosted RapidAPI service and its AI extras are external products, not implemented here.

Main entry points: [`src/contact_scraper/aggregate.py`](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/aggregate.py) (Result aggregation), [`src/contact_scraper/api.py`](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/api.py) (Scrape API), [`frontend/src/utils/api.tsx`](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/utils/api.tsx) (Dashboard API client), [`src/contact_scraper/crawler.py`](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/crawler.py) (Site crawler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Dashboard"]
  node_dashboard["Scraper dashboard<br/>[scrapers.py]"]
  node_input["Website inputs<br/>[InputComponent.tsx]"]
  node_api_client["Dashboard API client<br/>[api.tsx]"]
end

subgraph group_runtime["Scraping runtime"]
  node_cli["Python entrypoint<br/>[main.py]"]
  node_scrape_api["Scrape API<br/>[api.py]"]
  node_crawler["Site crawler<br/>[crawler.py]"]
  node_fetcher["Page fetcher<br/>[fetcher.py]"]
  node_chrome_pool["Chrome pools<br/>[chrome_manager.py]"]
  node_driver["Patchright driver"]
  node_html_text["HTML text processing<br/>[html_text.py]"]
  node_result_json["Structured results<br/>[api.py]"]
end

subgraph group_extract["Extraction"]
  node_aggregate["Result aggregation<br/>[aggregate.py]"]
  node_emails["Email extraction<br/>[extract_emails.py]"]
  node_phones["Phone extraction<br/>[extract_phones.py]"]
  node_socials["Social extraction<br/>[extract_socials.py]"]
  node_tech["Technology detection<br/>[tech_detect.py]"]
  node_social_patterns["Social patterns<br/>[social_regexes.py]"]
end

subgraph group_taskui["Task interface"]
  node_task_list["Task list and filters"]
  node_task_detail["Task results<br/>[TaskComponent.tsx]"]
  node_export["Result export"]
end

node_user(("Website user"))
node_website(("Target websites"))

node_user -->|"enters websites"| node_input
node_input -.->|"submits task"| node_api_client
node_dashboard -->|"registers scraper"| node_scrape_api
node_cli -->|"calls scraper"| node_scrape_api
node_api_client -->|"requests tasks"| node_task_list
node_api_client -->|"requests results"| node_task_detail
node_website -.->|"provides pages"| node_crawler
node_scrape_api -.->|"runs crawl"| node_crawler
node_crawler -->|"fetches pages"| node_fetcher
node_fetcher -->|"leases browser"| node_chrome_pool
node_chrome_pool -->|"builds driver"| node_driver
node_fetcher -->|"uses text"| node_html_text
node_crawler -->|"uses text"| node_html_text
node_scrape_api -.->|"aggregates results"| node_aggregate
node_aggregate -.->|"extracts emails"| node_emails
node_aggregate -.->|"extracts phones"| node_phones
node_aggregate -.->|"extracts profiles"| node_socials
node_aggregate -.->|"detects stack"| node_tech
node_socials -->|"uses patterns"| node_social_patterns
node_scrape_api -.->|"returns records"| node_result_json
node_task_list -.->|"loads task data"| node_api_client
node_task_detail -->|"loads task results"| node_api_client
node_task_detail -->|"offers export"| node_export
node_user -->|"reviews tasks"| node_task_list

click node_cli "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/main.py"
click node_dashboard "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/backend/scrapers.py"
click node_input "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/InputComponent/InputComponent.tsx"
click node_api_client "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/utils/api.tsx"
click node_scrape_api "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/api.py"
click node_crawler "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/crawler.py"
click node_fetcher "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/fetcher.py"
click node_chrome_pool "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/chrome_manager.py"
click node_driver "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/patchright_driver.py"
click node_html_text "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/html_text.py"
click node_aggregate "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/aggregate.py"
click node_emails "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/extract_emails.py"
click node_phones "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/extract_phones.py"
click node_socials "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/extract_socials.py"
click node_tech "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/tech_detect.py"
click node_social_patterns "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/social_regexes.py"
click node_task_list "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/OutputComponent/OutputComponent.tsx"
click node_task_detail "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/TaskComponent/TaskComponent.tsx"
click node_export "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/DownloadStickyBar/DownloadStickyBar.tsx"
click node_result_json "https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/api.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_input,node_api_client,node_user toneBlue
class node_cli,node_scrape_api,node_crawler,node_fetcher,node_chrome_pool,node_driver,node_html_text,node_result_json toneAmber
class node_aggregate,node_emails,node_phones,node_socials,node_tech,node_social_patterns toneMint
class node_task_list,node_task_detail,node_export toneRose
class node_website toneIndigo
```

## Components

### Dashboard

- [Scraper dashboard](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/backend/scrapers.py): `backend/scrapers.py`
- [Website inputs](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/InputComponent/InputComponent.tsx): `frontend/src/components/InputComponent/InputComponent.tsx`
- [Dashboard API client](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/utils/api.tsx): `frontend/src/utils/api.tsx`

### Scraping runtime

- [Python entrypoint](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/main.py): `main.py`
- [Scrape API](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/api.py): `src/contact_scraper/api.py`
- [Site crawler](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/crawler.py): `src/contact_scraper/crawler.py`
- [Page fetcher](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/fetcher.py): `src/contact_scraper/fetcher.py`
- [Chrome pools](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/chrome_manager.py): `src/contact_scraper/chrome_manager.py`
- [Patchright driver](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/patchright_driver.py): `src/contact_scraper/patchright_driver.py`
- [HTML text processing](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/html_text.py): `src/contact_scraper/html_text.py`
- [Structured results](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/api.py): `src/contact_scraper/api.py`

### Extraction

- [Result aggregation](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/aggregate.py): `src/contact_scraper/aggregate.py`
- [Email extraction](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/extract_emails.py): `src/contact_scraper/extract_emails.py`
- [Phone extraction](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/extract_phones.py): `src/contact_scraper/extract_phones.py`
- [Social extraction](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/extract_socials.py): `src/contact_scraper/extract_socials.py`
- [Technology detection](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/tech_detect.py): `src/contact_scraper/tech_detect.py`
- [Social patterns](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/src/contact_scraper/social_regexes.py): `src/contact_scraper/social_regexes.py`

### Task interface

- [Task list and filters](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/OutputComponent/OutputComponent.tsx): `frontend/src/components/OutputComponent/OutputComponent.tsx`
- [Task results](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/TaskComponent/TaskComponent.tsx): `frontend/src/components/TaskComponent/TaskComponent.tsx`
- [Result export](https://github.com/omkarcloud/website-email-contact-scraper/blob/master/frontend/src/components/DownloadStickyBar/DownloadStickyBar.tsx): `frontend/src/components/DownloadStickyBar/DownloadStickyBar.tsx`

### Other components

- **Website user**
- **Target websites**

## Connections

- Website user → Website inputs: enters websites (evidence: `README.md`)
- Website inputs → Dashboard API client: submits task (evidence: `frontend/src/components/InputComponent/InputComponent.tsx`)
- Scraper dashboard → Scrape API: registers scraper (evidence: `backend/scrapers.py`)
- Python entrypoint → Scrape API: calls scraper (evidence: `main.py`)
- Dashboard API client → Task list and filters: requests tasks (evidence: `frontend/src/utils/api.tsx`)
- Dashboard API client → Task results: requests results (evidence: `frontend/src/utils/api.tsx`)
- Target websites → Site crawler: provides pages
- Scrape API → Site crawler: runs crawl
- Site crawler → Page fetcher: fetches pages (evidence: `src/contact_scraper/crawler.py`)
- Page fetcher → Chrome pools: leases browser (evidence: `src/contact_scraper/fetcher.py`)
- Chrome pools → Patchright driver: builds driver (evidence: `src/contact_scraper/chrome_manager.py`)
- Page fetcher → HTML text processing: uses text (evidence: `src/contact_scraper/fetcher.py`)
- Site crawler → HTML text processing: uses text (evidence: `src/contact_scraper/crawler.py`)
- Scrape API → Result aggregation: aggregates results
- Result aggregation → Email extraction: extracts emails
- Result aggregation → Phone extraction: extracts phones
- Result aggregation → Social extraction: extracts profiles
- Result aggregation → Technology detection: detects stack
- Social extraction → Social patterns: uses patterns (evidence: `src/contact_scraper/extract_socials.py`)
- Scrape API → Structured results: returns records
- Task list and filters → Dashboard API client: loads task data (evidence: `frontend/src/components/OutputComponent/OutputComponent.tsx`)
- Task results → Dashboard API client: loads task results (evidence: `frontend/src/components/TaskComponent/TaskComponent.tsx`)
- Task results → Result export: offers export (evidence: `frontend/src/components/TaskComponent/TaskComponent.tsx`)
- Website user → Task list and filters: reviews tasks (evidence: `README.md`)
