# emrezdemir/crawler-boy architecture

> Architecture diagram of the GitHub repository emrezdemir/crawler-boy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/emrezdemir/crawler-boy
- Repository: https://github.com/emrezdemir/crawler-boy

## Overview

CrawlerBoy is an Electron desktop crawler and site downloader. Users configure a target and crawl options in the dashboard, start a crawl, monitor live progress and findings, then inspect or export saved results. The crawl engine coordinates URL traversal, robots policy, fetching, analysis, and downloads; worker threads handle parsing and passive recon. The README documents additional capabilities such as scope/filter controls, sitemap discovery, security audits, and export formats. Some UI and orchestration details are unsampled, so only documented or indexed relationships are shown.

Main entry points: [`src/main/crawler/CrawlEngine.js`](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/CrawlEngine.js) (Crawl engine), [`src/main/main.js`](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/main.js) (Electron main), [`src/main/crawler/Fetcher.js`](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Fetcher.js) (Page fetcher), [`src/main/crawler/utils.js`](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/utils.js) (URL and scope rules)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_desktop["Desktop experience"]
  node_dashboard["Crawl dashboard<br/>[renderer.js]"]
  node_bridge["Preload bridge<br/>[preload.js]"]
  node_app["Electron main<br/>[main.js]"]
end

subgraph group_control["Crawl control"]
  node_engine["Crawl engine<br/>[CrawlEngine.js]"]
  node_frontier["URL frontier<br/>[Frontier.js]"]
  node_robots["Robots policy<br/>[RobotsManager.js]"]
  node_utils["URL and scope rules<br/>[utils.js]"]
end

subgraph group_analysis["Retrieval and analysis"]
  node_fetcher["Page fetcher<br/>[Fetcher.js]"]
  node_analyzer["Analyzer pool<br/>[AnalyzerPool.js]"]
  node_worker["Analysis worker<br/>[analyzer-worker.js]"]
  node_parser["HTML parser<br/>[Parser.js]"]
  node_recon["Passive recon<br/>[Recon.js]"]
end

subgraph group_outputs["Persistence and results"]
  node_downloader["Asset and page saver<br/>[Downloader.js]"]
  node_session[("Crawl session files")]
  node_export["Result export<br/>[Downloader.js]"]
end

node_user(("Crawler operator"))
node_web(("Target websites"))

node_user -->|"configures crawl"| node_dashboard
node_dashboard -->|"uses API"| node_bridge
node_bridge -->|"IPC requests"| node_app
node_app -->|"loads window"| node_dashboard
node_app -->|"starts crawl"| node_engine
node_engine -->|"queues URLs"| node_frontier
node_engine -->|"checks policy"| node_robots
node_engine -->|"fetches pages"| node_fetcher
node_fetcher -->|"requests pages"| node_web
node_engine -->|"submits analysis"| node_analyzer
node_analyzer -->|"dispatches tasks"| node_worker
node_worker -->|"parses HTML"| node_parser
node_worker -->|"extracts findings"| node_recon
node_engine -->|"saves crawl data"| node_downloader
node_downloader -->|"fetches assets"| node_fetcher
node_downloader -->|"writes files"| node_session
node_app -->|"exports results"| node_export
node_export -->|"provides exports"| node_user
node_engine -->|"uses URL rules"| node_utils
node_frontier -->|"normalizes URLs"| node_utils
node_parser -->|"normalizes links"| node_utils
node_fetcher -->|"uses fetch helpers"| node_utils
node_robots -->|"reads robots.txt"| node_web

click node_dashboard "https://github.com/emrezdemir/crawler-boy/blob/main/src/renderer/renderer.js"
click node_bridge "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/preload.js"
click node_app "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/main.js"
click node_engine "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/CrawlEngine.js"
click node_frontier "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Frontier.js"
click node_robots "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/RobotsManager.js"
click node_utils "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/utils.js"
click node_fetcher "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Fetcher.js"
click node_analyzer "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/AnalyzerPool.js"
click node_worker "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/analyzer-worker.js"
click node_parser "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Parser.js"
click node_recon "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Recon.js"
click node_downloader "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Downloader.js"
click node_export "https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Downloader.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_bridge,node_app toneBlue
class node_engine,node_frontier,node_robots,node_utils toneAmber
class node_fetcher,node_analyzer,node_worker,node_parser,node_recon toneMint
class node_downloader,node_session,node_export toneRose
class node_user,node_web toneIndigo
```

## Components

### Desktop experience

- [Crawl dashboard](https://github.com/emrezdemir/crawler-boy/blob/main/src/renderer/renderer.js): `src/renderer/renderer.js`
- [Preload bridge](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/preload.js): `src/main/preload.js`
- [Electron main](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/main.js): `src/main/main.js`

### Crawl control

- [Crawl engine](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/CrawlEngine.js): `src/main/crawler/CrawlEngine.js`
- [URL frontier](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Frontier.js): `src/main/crawler/Frontier.js`
- [Robots policy](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/RobotsManager.js): `src/main/crawler/RobotsManager.js`
- [URL and scope rules](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/utils.js): `src/main/crawler/utils.js`

### Retrieval and analysis

- [Page fetcher](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Fetcher.js): `src/main/crawler/Fetcher.js`
- [Analyzer pool](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/AnalyzerPool.js): `src/main/crawler/AnalyzerPool.js`
- [Analysis worker](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/analyzer-worker.js): `src/main/crawler/analyzer-worker.js`
- [HTML parser](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Parser.js): `src/main/crawler/Parser.js`
- [Passive recon](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Recon.js): `src/main/crawler/Recon.js`

### Persistence and results

- [Asset and page saver](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Downloader.js): `src/main/crawler/Downloader.js`
- **Crawl session files**
- [Result export](https://github.com/emrezdemir/crawler-boy/blob/main/src/main/crawler/Downloader.js): `src/main/crawler/Downloader.js`

### Other components

- **Crawler operator**
- **Target websites**

## Connections

- Crawler operator → Crawl dashboard: configures crawl (evidence: `README.md`)
- Crawl dashboard → Preload bridge: uses API (evidence: `README.md`)
- Preload bridge → Electron main: IPC requests (evidence: `README.md`)
- Electron main → Crawl dashboard: loads window (evidence: `src/main/main.js`)
- Electron main → Crawl engine: starts crawl (evidence: `src/main/main.js`)
- Crawl engine → URL frontier: queues URLs (evidence: `src/main/crawler/CrawlEngine.js`)
- Crawl engine → Robots policy: checks policy (evidence: `src/main/crawler/CrawlEngine.js`)
- Crawl engine → Page fetcher: fetches pages (evidence: `README.md`)
- Page fetcher → Target websites: requests pages (evidence: `README.md`)
- Crawl engine → Analyzer pool: submits analysis (evidence: `src/main/crawler/CrawlEngine.js`)
- Analyzer pool → Analysis worker: dispatches tasks (evidence: `src/main/crawler/AnalyzerPool.js`)
- Analysis worker → HTML parser: parses HTML (evidence: `src/main/crawler/analyzer-worker.js`)
- Analysis worker → Passive recon: extracts findings (evidence: `src/main/crawler/analyzer-worker.js`)
- Crawl engine → Asset and page saver: saves crawl data (evidence: `src/main/crawler/CrawlEngine.js`)
- Asset and page saver → Page fetcher: fetches assets (evidence: `src/main/crawler/Downloader.js`)
- Asset and page saver → Crawl session files: writes files (evidence: `README.md`)
- Electron main → Result export: exports results (evidence: `src/main/main.js`)
- Result export → Crawler operator: provides exports (evidence: `README.md`)
- Crawl engine → URL and scope rules: uses URL rules (evidence: `src/main/crawler/CrawlEngine.js`)
- URL frontier → URL and scope rules: normalizes URLs (evidence: `src/main/crawler/Frontier.js`)
- HTML parser → URL and scope rules: normalizes links (evidence: `src/main/crawler/Parser.js`)
- Page fetcher → URL and scope rules: uses fetch helpers (evidence: `src/main/crawler/Fetcher.js`)
- Robots policy → Target websites: reads robots.txt (evidence: `README.md`)