How emrezdemir/crawler-boy works
CrawlerBoy is an Electron desktop crawler and site downloader. Users configure a target and crawl options in the dashboard, start a crawl, monitor live progress and findings, then inspect or export saved results. The crawl engine coordinates URL traversal, robots policy, fetching, analysis, and downloads; worker threads handle parsing and passive recon. The README documents additional capabilities such as scope/filter controls, sitemap discovery, security audits, and export formats. Some UI and orchestration details are unsampled, so only documented or indexed relationships are shown.
Main entry points: CrawlEngine.js Crawl engine · main.js Electron main · Fetcher.js Page fetcher · utils.js URL and scope rules
Components 17
Desktop experience
- Crawl dashboard
src/renderer/renderer.js - Preload bridge
src/main/preload.js - Electron main
src/main/main.js
Crawl control
- Crawl engine
src/main/crawler/CrawlEngine.js - URL frontier
src/main/crawler/Frontier.js - Robots policy
src/main/crawler/RobotsManager.js - URL and scope rules
src/main/crawler/utils.js
Retrieval and analysis
- Page fetcher
src/main/crawler/Fetcher.js - Analyzer pool
src/main/crawler/AnalyzerPool.js - Analysis worker
src/main/crawler/analyzer-worker.js - HTML parser
src/main/crawler/Parser.js - Passive recon
src/main/crawler/Recon.js
Persistence and results
- Asset and page saver
src/main/crawler/Downloader.js - Crawl session files
- Result export
src/main/crawler/Downloader.js
Other components
- Crawler operator
- Target websites
Connections 23
- Crawler operator → Crawl dashboard: configures crawl
README.md - Crawl dashboard → Preload bridge: uses API
README.md - Preload bridge → Electron main: IPC requests
README.md - Electron main → Crawl dashboard: loads window
src/main/main.js - Electron main → Crawl engine: starts crawl
src/main/main.js - Crawl engine → URL frontier: queues URLs
src/main/crawler/CrawlEngine.js - Crawl engine → Robots policy: checks policy
src/main/crawler/CrawlEngine.js - Crawl engine → Page fetcher: fetches pages
README.md - Page fetcher → Target websites: requests pages
README.md - Crawl engine → Analyzer pool: submits analysis
src/main/crawler/CrawlEngine.js - Analyzer pool → Analysis worker: dispatches tasks
src/main/crawler/AnalyzerPool.js - Analysis worker → HTML parser: parses HTML
src/main/crawler/analyzer-worker.js - Analysis worker → Passive recon: extracts findings
src/main/crawler/analyzer-worker.js - Crawl engine → Asset and page saver: saves crawl data
src/main/crawler/CrawlEngine.js - Asset and page saver → Page fetcher: fetches assets
src/main/crawler/Downloader.js - Asset and page saver → Crawl session files: writes files
README.md - Electron main → Result export: exports results
src/main/main.js - Result export → Crawler operator: provides exports
README.md - Crawl engine → URL and scope rules: uses URL rules
src/main/crawler/CrawlEngine.js - URL frontier → URL and scope rules: normalizes URLs
src/main/crawler/Frontier.js - HTML parser → URL and scope rules: normalizes links
src/main/crawler/Parser.js - Page fetcher → URL and scope rules: uses fetch helpers
src/main/crawler/Fetcher.js - Robots policy → Target websites: reads robots.txt
README.md