# sujaljaiswal25/go-fastest-scraper architecture

> Architecture diagram of the GitHub repository sujaljaiswal25/go-fastest-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sujaljaiswal25/go-fastest-scraper
- Repository: https://github.com/sujaljaiswal25/go-fastest-scraper

## Overview

Go Scraper is a CLI web scraper and crawler. Users submit URLs for single or batch scraping, BFS crawling, sitemap scraping, or sitemap discovery. The main runtime dispatches those workflows through an orchestrator; pages are rate-limited, fetched with progressive HTTP/browser escalation, extracted into content and structured metadata, then cached and written as requested artifacts. The README also documents a reusable Chrome daemon and cache controls. The tree omits the imported output package, so its implementation path and internal details cannot be verified; output relationships below are based on the orchestrator excerpt and README.

Main entry points: [`internal/fetch/pipeline.go`](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/pipeline.go) (Escalation pipeline), [`internal/orchestrator/orchestrator.go`](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/orchestrator/orchestrator.go) (Workflow orchestrator), [`internal/crawl/bfs.go`](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/bfs.go) (BFS crawler), [`cmd/scraper/main.go`](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/cmd/scraper/main.go) (CLI commands)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI and orchestration"]
  node_cli["CLI commands<br/>[main.go]"]
  node_config["Run configuration<br/>[config.go]"]
  node_orchestrator["Workflow orchestrator<br/>[orchestrator.go]"]
  node_worker["Scrape worker pool<br/>[pool.go]"]
  node_limiter["Domain limiter<br/>[limiter.go]"]
end

subgraph group_discovery["Crawl and discovery"]
  node_bfs["BFS crawler<br/>[bfs.go]"]
  node_robots["Robots rules<br/>[robots.go]"]
  node_sitemap["Sitemap discovery<br/>[sitemap.go]"]
end

subgraph group_retrieval["Fetch and resilience"]
  node_fetch["Escalation pipeline<br/>[pipeline.go]"]
  node_http["HTTP fetch<br/>[http.go]"]
  node_stealth["Stealth browser fetch<br/>[stealth.go]"]
  node_browser["Browser pool and daemon<br/>[browser_pool.go]"]
  node_site_detect["Site type detection<br/>[site_type.go]"]
end

subgraph group_content["Extraction and results"]
  node_cache[("Result cache<br/>[cache.go]")]
  node_extract["Page extraction"]
  node_markdown["Markdown conversion<br/>[markdown.go]"]
  node_url_rewrite["URL rewriting<br/>[url_rewrite.go]"]
  node_artifacts["Output artifacts"]
end

node_user(("CLI user"))
node_website(("Target websites"))

node_user -->|"runs commands"| node_cli
node_cli -->|"merges settings"| node_config
node_cli -->|"starts workflows"| node_orchestrator
node_orchestrator -->|"dispatches jobs"| node_worker
node_orchestrator -->|"starts crawl"| node_bfs
node_orchestrator -->|"discovers URLs"| node_sitemap
node_orchestrator -->|"waits per domain"| node_limiter
node_worker -->|"processes jobs"| node_fetch
node_bfs -->|"checks crawl rules"| node_robots
node_bfs -->|"waits per domain"| node_limiter
node_bfs -->|"fetches pages"| node_fetch
node_fetch -->|"checks and stores"| node_cache
node_fetch -->|"rewrites URL"| node_url_rewrite
node_fetch -->|"probes page"| node_http
node_fetch -->|"escalates fetch"| node_stealth
node_fetch -->|"checks content"| node_site_detect
node_stealth -.->|"uses browser pool"| node_browser
node_http -.->|"requests pages"| node_website
node_stealth -.->|"renders pages"| node_website
node_fetch -->|"extracts page data"| node_extract
node_fetch -->|"builds Markdown"| node_markdown
node_orchestrator -->|"writes results"| node_artifacts
node_bfs -->|"returns crawl results"| node_orchestrator

click node_cli "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/cmd/scraper/main.go"
click node_config "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/config/config.go"
click node_orchestrator "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/orchestrator/orchestrator.go"
click node_worker "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/worker/pool.go"
click node_limiter "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/ratelimit/limiter.go"
click node_bfs "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/bfs.go"
click node_robots "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/robots.go"
click node_sitemap "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/sitemap.go"
click node_fetch "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/pipeline.go"
click node_http "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/http.go"
click node_stealth "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/stealth.go"
click node_browser "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/browser_pool.go"
click node_site_detect "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/detect/site_type.go"
click node_cache "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/cache/cache.go"
click node_extract "https://github.com/sujaljaiswal25/go-fastest-scraper/tree/main/internal/extract"
click node_markdown "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/convert/markdown.go"
click node_url_rewrite "https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/convert/url_rewrite.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_orchestrator,node_worker,node_limiter,node_user toneBlue
class node_bfs,node_robots,node_sitemap toneAmber
class node_fetch,node_http,node_stealth,node_browser,node_site_detect toneMint
class node_cache,node_extract,node_markdown,node_url_rewrite,node_artifacts toneRose
class node_website toneIndigo
```

## Components

### CLI and orchestration

- [CLI commands](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/cmd/scraper/main.go): `cmd/scraper/main.go`
- [Run configuration](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/config/config.go): `internal/config/config.go`
- [Workflow orchestrator](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/orchestrator/orchestrator.go): `internal/orchestrator/orchestrator.go`
- [Scrape worker pool](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/worker/pool.go): `internal/worker/pool.go`
- [Domain limiter](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/ratelimit/limiter.go): `internal/ratelimit/limiter.go`

### Crawl and discovery

- [BFS crawler](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/bfs.go): `internal/crawl/bfs.go`
- [Robots rules](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/robots.go): `internal/crawl/robots.go`
- [Sitemap discovery](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/crawl/sitemap.go): `internal/crawl/sitemap.go`

### Fetch and resilience

- [Escalation pipeline](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/pipeline.go): `internal/fetch/pipeline.go`
- [HTTP fetch](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/http.go): `internal/fetch/http.go`
- [Stealth browser fetch](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/stealth.go): `internal/fetch/stealth.go`
- [Browser pool and daemon](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/fetch/browser_pool.go): `internal/fetch/browser_pool.go`
- [Site type detection](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/detect/site_type.go): `internal/detect/site_type.go`

### Extraction and results

- [Result cache](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/cache/cache.go): `internal/cache/cache.go`
- [Page extraction](https://github.com/sujaljaiswal25/go-fastest-scraper/tree/main/internal/extract): `internal/extract`
- [Markdown conversion](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/convert/markdown.go): `internal/convert/markdown.go`
- [URL rewriting](https://github.com/sujaljaiswal25/go-fastest-scraper/blob/main/internal/convert/url_rewrite.go): `internal/convert/url_rewrite.go`
- **Output artifacts**

### Other components

- **CLI user**
- **Target websites**

## Connections

- CLI user → CLI commands: runs commands (evidence: `README.md`)
- CLI commands → Run configuration: merges settings (evidence: `cmd/scraper/main.go`)
- CLI commands → Workflow orchestrator: starts workflows (evidence: `cmd/scraper/main.go`)
- Workflow orchestrator → Scrape worker pool: dispatches jobs (evidence: `internal/orchestrator/orchestrator.go`)
- Workflow orchestrator → BFS crawler: starts crawl (evidence: `internal/orchestrator/orchestrator.go`)
- Workflow orchestrator → Sitemap discovery: discovers URLs (evidence: `internal/orchestrator/orchestrator.go`)
- Workflow orchestrator → Domain limiter: waits per domain (evidence: `internal/orchestrator/orchestrator.go`)
- Scrape worker pool → Escalation pipeline: processes jobs
- BFS crawler → Robots rules: checks crawl rules (evidence: `internal/crawl/bfs.go`)
- BFS crawler → Domain limiter: waits per domain (evidence: `internal/crawl/bfs.go`)
- BFS crawler → Escalation pipeline: fetches pages (evidence: `internal/crawl/bfs.go`)
- Escalation pipeline → Result cache: checks and stores (evidence: `internal/fetch/pipeline.go`)
- Escalation pipeline → URL rewriting: rewrites URL (evidence: `internal/fetch/pipeline.go`)
- Escalation pipeline → HTTP fetch: probes page (evidence: `internal/fetch/pipeline.go`)
- Escalation pipeline → Stealth browser fetch: escalates fetch (evidence: `internal/fetch/pipeline.go`)
- Escalation pipeline → Site type detection: checks content (evidence: `internal/fetch/pipeline.go`)
- Stealth browser fetch → Browser pool and daemon: uses browser pool
- HTTP fetch → Target websites: requests pages
- Stealth browser fetch → Target websites: renders pages
- Escalation pipeline → Page extraction: extracts page data (evidence: `internal/fetch/pipeline.go`)
- Escalation pipeline → Markdown conversion: builds Markdown (evidence: `internal/fetch/pipeline.go`)
- Workflow orchestrator → Output artifacts: writes results (evidence: `internal/orchestrator/orchestrator.go`)
- BFS crawler → Workflow orchestrator: returns crawl results (evidence: `internal/orchestrator/orchestrator.go`)
