How omkarcloud/website-email-contact-scraper works
This repository provides a website-contact scraper with both a Python callable/CLI and a local browser dashboard. The main workflow accepts website URLs and a crawl mode, crawls pages, extracts contact details and technologies, aggregates structured results, and presents task results for filtering and export. README-listed extraction capabilities are represented, but many internals are unsampled; edges are limited to relationships supported by the excerpts, SOURCE INDEX references, or README. The hosted RapidAPI service and its AI extras are external products, not implemented here.
Main entry points: aggregate.py Result aggregation · api.py Scrape API · api.tsx Dashboard API client · crawler.py Site crawler
Components 22
Dashboard
- Scraper dashboard
backend/scrapers.py - Website inputs
frontend/src/components/InputComponent/InputComponent.tsx - Dashboard API client
frontend/src/utils/api.tsx
Scraping runtime
- Python entrypoint
main.py - Scrape API
src/contact_scraper/api.py - Site crawler
src/contact_scraper/crawler.py - Page fetcher
src/contact_scraper/fetcher.py - Chrome pools
src/contact_scraper/chrome_manager.py - Patchright driver
src/contact_scraper/patchright_driver.py - HTML text processing
src/contact_scraper/html_text.py - Structured results
src/contact_scraper/api.py
Extraction
- Result aggregation
src/contact_scraper/aggregate.py - Email extraction
src/contact_scraper/extract_emails.py - Phone extraction
src/contact_scraper/extract_phones.py - Social extraction
src/contact_scraper/extract_socials.py - Technology detection
src/contact_scraper/tech_detect.py - Social patterns
src/contact_scraper/social_regexes.py
Task interface
- Task list and filters
frontend/src/components/OutputComponent/OutputComponent.tsx - Task results
frontend/src/components/TaskComponent/TaskComponent.tsx - Result export
frontend/src/components/DownloadStickyBar/DownloadStickyBar.tsx
Other components
- Website user
- Target websites
Connections 24
- Website user → Website inputs: enters websites
README.md - Website inputs → Dashboard API client: submits task
frontend/src/components/InputComponent/InputComponent.tsx - Scraper dashboard → Scrape API: registers scraper
backend/scrapers.py - Python entrypoint → Scrape API: calls scraper
main.py - Dashboard API client → Task list and filters: requests tasks
frontend/src/utils/api.tsx - Dashboard API client → Task results: requests results
frontend/src/utils/api.tsx - Target websites → Site crawler: provides pages
- Scrape API → Site crawler: runs crawl
- Site crawler → Page fetcher: fetches pages
src/contact_scraper/crawler.py - Page fetcher → Chrome pools: leases browser
src/contact_scraper/fetcher.py - Chrome pools → Patchright driver: builds driver
src/contact_scraper/chrome_manager.py - Page fetcher → HTML text processing: uses text
src/contact_scraper/fetcher.py - Site crawler → HTML text processing: uses text
src/contact_scraper/crawler.py - Scrape API → Result aggregation: aggregates results
- Result aggregation → Email extraction: extracts emails
- Result aggregation → Phone extraction: extracts phones
- Result aggregation → Social extraction: extracts profiles
- Result aggregation → Technology detection: detects stack
- Social extraction → Social patterns: uses patterns
src/contact_scraper/extract_socials.py - Scrape API → Structured results: returns records
- Task list and filters → Dashboard API client: loads task data
frontend/src/components/OutputComponent/OutputComponent.tsx - Task results → Dashboard API client: loads task results
frontend/src/components/TaskComponent/TaskComponent.tsx - Task results → Result export: offers export
frontend/src/components/TaskComponent/TaskComponent.tsx - Website user → Task list and filters: reviews tasks
README.md