# gocolly/colly architecture

> Architecture diagram of the GitHub repository gocolly/colly, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gocolly/colly
- Repository: https://github.com/gocolly/colly

## Overview

Colly is a Go crawling and scraping framework. A caller configures a Collector and starts a visit; Colly fetches pages, turns responses into HTML/XML representations, and invokes registered callbacks to extract data or schedule further visits. The request queue supports concurrent crawling, while storage tracks visited requests and cookies. README capabilities also include caching, robots.txt, per-domain delay/concurrency, distributed storage, environment configuration, and extensions. Those subsystems are represented at their documented source locations; their detailed wiring is not inferred where source excerpts were unavailable.

Main entry points: [`colly.go`](https://github.com/gocolly/colly/blob/master/colly.go) (Collector API), [`http_backend.go`](https://github.com/gocolly/colly/blob/master/http_backend.go) (HTTP backend), [`request.go`](https://github.com/gocolly/colly/blob/master/request.go) (Request model), [`unmarshal.go`](https://github.com/gocolly/colly/blob/master/unmarshal.go) (Response decoding)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawl["Crawl orchestration"]
  node_collector["Collector API<br/>[colly.go]"]
  node_request["Request model<br/>[request.go]"]
  node_context["Request context<br/>[context.go]"]
  node_queue["Concurrent queue<br/>[queue.go]"]
end

subgraph group_fetch["Request and response"]
  node_backend["HTTP backend<br/>[http_backend.go]"]
  node_response["Response model<br/>[response.go]"]
  node_trace["HTTP tracing<br/>[http_trace.go]"]
end

subgraph group_parse["Parsing and callbacks"]
  node_html["HTML elements<br/>[htmlelement.go]"]
  node_xml["XML elements<br/>[xmlelement.go]"]
  node_unmarshal["Response decoding<br/>[unmarshal.go]"]
  node_callbacks["Scraper callbacks<br/>[colly.go]"]
end

subgraph group_state["State and integrations"]
  node_storage[("Visited and cookies<br/>[storage.go]")]
  node_cache[("Response cache<br/>[colly.go]")]
  node_robots["Robots rules<br/>[colly.go]"]
  node_limits["Domain limits<br/>[colly.go]"]
  node_proxy["Proxy switching<br/>[proxy.go]"]
  node_extensions["Request extensions<br/>[extensions.go]"]
  node_debugger["Debug events<br/>[debug.go]"]
  node_webdebug["Web debugger<br/>[webdebugger.go]"]
  node_distributed[("Custom backends<br/>[storage.go]")]
end

node_caller(("Scraper caller"))
node_website(("Target website"))

node_caller -->|"configures and visits"| node_collector
node_collector -->|"creates requests"| node_request
node_collector -->|"checks and records"| node_storage
node_collector -->|"dispatches requests"| node_backend
node_backend -->|"sends HTTP"| node_website
node_website -->|"returns response"| node_backend
node_backend -->|"builds response"| node_response
node_response -->|"decodes body"| node_unmarshal
node_unmarshal -->|"provides HTML"| node_html
node_unmarshal -->|"provides XML"| node_xml
node_collector -->|"invokes handlers"| node_callbacks
node_callbacks -->|"schedules visits"| node_request
node_request -->|"carries callback data"| node_context
node_queue -->|"dispatches queued work"| node_request
node_queue -->|"runs requests"| node_collector
node_collector -.->|"caches responses"| node_cache
node_collector -.->|"checks policy"| node_robots
node_collector -.->|"controls request rate"| node_limits
node_backend -.->|"selects proxy"| node_proxy
node_extensions -.->|"configures behavior"| node_collector
node_collector -.->|"emits events"| node_debugger
node_debugger -.->|"updates status"| node_webdebug
node_collector -.->|"uses custom storage"| node_distributed
node_collector -.->|"records HTTP trace"| node_trace

click node_collector "https://github.com/gocolly/colly/blob/master/colly.go"
click node_request "https://github.com/gocolly/colly/blob/master/request.go"
click node_context "https://github.com/gocolly/colly/blob/master/context.go"
click node_queue "https://github.com/gocolly/colly/blob/master/queue/queue.go"
click node_backend "https://github.com/gocolly/colly/blob/master/http_backend.go"
click node_response "https://github.com/gocolly/colly/blob/master/response.go"
click node_trace "https://github.com/gocolly/colly/blob/master/http_trace.go"
click node_html "https://github.com/gocolly/colly/blob/master/htmlelement.go"
click node_xml "https://github.com/gocolly/colly/blob/master/xmlelement.go"
click node_unmarshal "https://github.com/gocolly/colly/blob/master/unmarshal.go"
click node_callbacks "https://github.com/gocolly/colly/blob/master/colly.go"
click node_storage "https://github.com/gocolly/colly/blob/master/storage/storage.go"
click node_cache "https://github.com/gocolly/colly/blob/master/colly.go"
click node_robots "https://github.com/gocolly/colly/blob/master/colly.go"
click node_limits "https://github.com/gocolly/colly/blob/master/colly.go"
click node_proxy "https://github.com/gocolly/colly/blob/master/proxy/proxy.go"
click node_extensions "https://github.com/gocolly/colly/blob/master/extensions/extensions.go"
click node_debugger "https://github.com/gocolly/colly/blob/master/debug/debug.go"
click node_webdebug "https://github.com/gocolly/colly/blob/master/debug/webdebugger.go"
click node_distributed "https://github.com/gocolly/colly/blob/master/storage/storage.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collector,node_request,node_context,node_queue toneBlue
class node_backend,node_response,node_trace toneAmber
class node_html,node_xml,node_unmarshal,node_callbacks toneMint
class node_storage,node_cache,node_robots,node_limits,node_proxy,node_extensions,node_debugger,node_webdebug,node_distributed toneRose
class node_caller,node_website toneIndigo
```

## Components

### Crawl orchestration

- [Collector API](https://github.com/gocolly/colly/blob/master/colly.go): `colly.go`
- [Request model](https://github.com/gocolly/colly/blob/master/request.go): `request.go`
- [Request context](https://github.com/gocolly/colly/blob/master/context.go): `context.go`
- [Concurrent queue](https://github.com/gocolly/colly/blob/master/queue/queue.go): `queue/queue.go`

### Request and response

- [HTTP backend](https://github.com/gocolly/colly/blob/master/http_backend.go): `http_backend.go`
- [Response model](https://github.com/gocolly/colly/blob/master/response.go): `response.go`
- [HTTP tracing](https://github.com/gocolly/colly/blob/master/http_trace.go): `http_trace.go`

### Parsing and callbacks

- [HTML elements](https://github.com/gocolly/colly/blob/master/htmlelement.go): `htmlelement.go`
- [XML elements](https://github.com/gocolly/colly/blob/master/xmlelement.go): `xmlelement.go`
- [Response decoding](https://github.com/gocolly/colly/blob/master/unmarshal.go): `unmarshal.go`
- [Scraper callbacks](https://github.com/gocolly/colly/blob/master/colly.go): `colly.go`

### State and integrations

- [Visited and cookies](https://github.com/gocolly/colly/blob/master/storage/storage.go): `storage/storage.go`
- [Response cache](https://github.com/gocolly/colly/blob/master/colly.go): `colly.go`
- [Robots rules](https://github.com/gocolly/colly/blob/master/colly.go): `colly.go`
- [Domain limits](https://github.com/gocolly/colly/blob/master/colly.go): `colly.go`
- [Proxy switching](https://github.com/gocolly/colly/blob/master/proxy/proxy.go): `proxy/proxy.go`
- [Request extensions](https://github.com/gocolly/colly/blob/master/extensions/extensions.go): `extensions/extensions.go`
- [Debug events](https://github.com/gocolly/colly/blob/master/debug/debug.go): `debug/debug.go`
- [Web debugger](https://github.com/gocolly/colly/blob/master/debug/webdebugger.go): `debug/webdebugger.go`
- [Custom backends](https://github.com/gocolly/colly/blob/master/storage/storage.go): `storage/storage.go`

### Other components

- **Scraper caller**
- **Target website**

## Connections

- Scraper caller → Collector API: configures and visits
- Collector API → Request model: creates requests
- Collector API → Visited and cookies: checks and records
- Collector API → HTTP backend: dispatches requests
- HTTP backend → Target website: sends HTTP
- Target website → HTTP backend: returns response
- HTTP backend → Response model: builds response
- Response model → Response decoding: decodes body
- Response decoding → HTML elements: provides HTML
- Response decoding → XML elements: provides XML
- Collector API → Scraper callbacks: invokes handlers
- Scraper callbacks → Request model: schedules visits
- Request model → Request context: carries callback data
- Concurrent queue → Request model: dispatches queued work
- Concurrent queue → Collector API: runs requests
- Collector API → Response cache: caches responses
- Collector API → Robots rules: checks policy
- Collector API → Domain limits: controls request rate
- HTTP backend → Proxy switching: selects proxy
- Request extensions → Collector API: configures behavior
- Collector API → Debug events: emits events
- Debug events → Web debugger: updates status
- Collector API → Custom backends: uses custom storage
- Collector API → HTTP tracing: records HTTP trace