How gocolly/colly works
Colly is a Go crawling and scraping framework. A caller configures a Collector and starts a visit; Colly fetches pages, turns responses into HTML/XML representations, and invokes registered callbacks to extract data or schedule further visits. The request queue supports concurrent crawling, while storage tracks visited requests and cookies. README capabilities also include caching, robots.txt, per-domain delay/concurrency, distributed storage, environment configuration, and extensions. Those subsystems are represented at their documented source locations; their detailed wiring is not inferred where source excerpts were unavailable.
Main entry points: colly.go Collector API · http_backend.go HTTP backend · request.go Request model · unmarshal.go Response decoding
Components 22
Crawl orchestration
- Collector API
colly.go - Request model
request.go - Request context
context.go - Concurrent queue
queue/queue.go
Request and response
- HTTP backend
http_backend.go - Response model
response.go - HTTP tracing
http_trace.go
Parsing and callbacks
- HTML elements
htmlelement.go - XML elements
xmlelement.go - Response decoding
unmarshal.go - Scraper callbacks
colly.go
State and integrations
- Visited and cookies
storage/storage.go - Response cache
colly.go - Robots rules
colly.go - Domain limits
colly.go - Proxy switching
proxy/proxy.go - Request extensions
extensions/extensions.go - Debug events
debug/debug.go - Web debugger
debug/webdebugger.go - Custom backends
storage/storage.go
Other components
- Scraper caller
- Target website
Connections 24
- Scraper caller → Collector API: configures and visits
- Collector API → Request model: creates requests
- Collector API → Visited and cookies: checks and records
- Collector API → HTTP backend: dispatches requests
- HTTP backend → Target website: sends HTTP
- Target website → HTTP backend: returns response
- HTTP backend → Response model: builds response
- Response model → Response decoding: decodes body
- Response decoding → HTML elements: provides HTML
- Response decoding → XML elements: provides XML
- Collector API → Scraper callbacks: invokes handlers
- Scraper callbacks → Request model: schedules visits
- Request model → Request context: carries callback data
- Concurrent queue → Request model: dispatches queued work
- Concurrent queue → Collector API: runs requests
- Collector API → Response cache: caches responses
- Collector API → Robots rules: checks policy
- Collector API → Domain limits: controls request rate
- HTTP backend → Proxy switching: selects proxy
- Request extensions → Collector API: configures behavior
- Collector API → Debug events: emits events
- Debug events → Web debugger: updates status
- Collector API → Custom backends: uses custom storage
- Collector API → HTTP tracing: records HTTP trace