How sujaljaiswal25/go-fastest-scraper works
Go Scraper is a CLI web scraper and crawler. Users submit URLs for single or batch scraping, BFS crawling, sitemap scraping, or sitemap discovery. The main runtime dispatches those workflows through an orchestrator; pages are rate-limited, fetched with progressive HTTP/browser escalation, extracted into content and structured metadata, then cached and written as requested artifacts. The README also documents a reusable Chrome daemon and cache controls. The tree omits the imported output package, so its implementation path and internal details cannot be verified; output relationships below are based on the orchestrator excerpt and README.
Main entry points: pipeline.go Escalation pipeline · orchestrator.go Workflow orchestrator · bfs.go BFS crawler · main.go CLI commands
Components 20
CLI and orchestration
- CLI commands
cmd/scraper/main.go - Run configuration
internal/config/config.go - Workflow orchestrator
internal/orchestrator/orchestrator.go - Scrape worker pool
internal/worker/pool.go - Domain limiter
internal/ratelimit/limiter.go
Crawl and discovery
- BFS crawler
internal/crawl/bfs.go - Robots rules
internal/crawl/robots.go - Sitemap discovery
internal/crawl/sitemap.go
Fetch and resilience
- Escalation pipeline
internal/fetch/pipeline.go - HTTP fetch
internal/fetch/http.go - Stealth browser fetch
internal/fetch/stealth.go - Browser pool and daemon
internal/fetch/browser_pool.go - Site type detection
internal/detect/site_type.go
Extraction and results
- Result cache
internal/cache/cache.go - Page extraction
internal/extract - Markdown conversion
internal/convert/markdown.go - URL rewriting
internal/convert/url_rewrite.go - Output artifacts
Other components
- CLI user
- Target websites
Connections 23
- CLI user → CLI commands: runs commands
README.md - CLI commands → Run configuration: merges settings
cmd/scraper/main.go - CLI commands → Workflow orchestrator: starts workflows
cmd/scraper/main.go - Workflow orchestrator → Scrape worker pool: dispatches jobs
internal/orchestrator/orchestrator.go - Workflow orchestrator → BFS crawler: starts crawl
internal/orchestrator/orchestrator.go - Workflow orchestrator → Sitemap discovery: discovers URLs
internal/orchestrator/orchestrator.go - Workflow orchestrator → Domain limiter: waits per domain
internal/orchestrator/orchestrator.go - Scrape worker pool → Escalation pipeline: processes jobs
- BFS crawler → Robots rules: checks crawl rules
internal/crawl/bfs.go - BFS crawler → Domain limiter: waits per domain
internal/crawl/bfs.go - BFS crawler → Escalation pipeline: fetches pages
internal/crawl/bfs.go - Escalation pipeline → Result cache: checks and stores
internal/fetch/pipeline.go - Escalation pipeline → URL rewriting: rewrites URL
internal/fetch/pipeline.go - Escalation pipeline → HTTP fetch: probes page
internal/fetch/pipeline.go - Escalation pipeline → Stealth browser fetch: escalates fetch
internal/fetch/pipeline.go - Escalation pipeline → Site type detection: checks content
internal/fetch/pipeline.go - Stealth browser fetch → Browser pool and daemon: uses browser pool
- HTTP fetch → Target websites: requests pages
- Stealth browser fetch → Target websites: renders pages
- Escalation pipeline → Page extraction: extracts page data
internal/fetch/pipeline.go - Escalation pipeline → Markdown conversion: builds Markdown
internal/fetch/pipeline.go - Workflow orchestrator → Output artifacts: writes results
internal/orchestrator/orchestrator.go - BFS crawler → Workflow orchestrator: returns crawl results
internal/orchestrator/orchestrator.go