# code4craft/webmagic architecture

> Architecture diagram of the GitHub repository code4craft/webmagic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/code4craft/webmagic
- Repository: https://github.com/code4craft/webmagic

## Overview

WebMagic is a Java web-crawling framework covering URL management, downloading, extraction, and persistence. A user configures a Spider with a page processor and starting URLs; the crawler schedules requests, downloads pages, extracts fields and follow-up URLs, then sends results to pipelines. Annotation-based POJO models provide an alternate extraction path. The tree and README document distributed scheduling and browser integrations; sampled source confirms pipeline behavior, but Spider’s internal dispatch is not sampled, so uncertain implementation-level links are omitted.

Main entry points: [`webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java`](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java) (Spider), [`webmagic-core/src/main/java/us/codecraft/webmagic/processor/PageProcessor.java`](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/processor/PageProcessor.java) (Page processor), [`webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPageProcessor.java`](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPageProcessor.java) (Model processor), [`webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java`](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java) (HTTP downloader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawl["Crawl runtime"]
  node_spider["Spider<br/>[Spider.java]"]
  node_request["Request<br/>[Request.java]"]
  node_downloader["HTTP downloader"]
  node_page["Page<br/>[Page.java]"]
  node_browser["Browser downloaders"]
  node_monitor["Spider monitor<br/>[SpiderMonitor.java]"]
end

subgraph group_extract["Extraction"]
  node_processor["Page processor<br/>[PageProcessor.java]"]
  node_selectors["Content selectors<br/>[Selectors.java]"]
  node_result["Result items<br/>[ResultItems.java]"]
  node_script_processor["Script processor"]
end

subgraph group_models["Model extensions"]
  node_model_processor["Model processor"]
  node_model_extractor["POJO extractor"]
  node_oo_spider["OOSpider<br/>[OOSpider.java]"]
end

subgraph group_outputs["Scheduling and output"]
  node_scheduler["Scheduler<br/>[Scheduler.java]"]
  node_redis_scheduler["Redis schedulers"]
  node_pipeline["Result pipeline<br/>[Pipeline.java]"]
  node_model_pipeline["Model pipeline<br/>[ModelPipeline.java]"]
  node_file_pipeline["File output<br/>[FilePipeline.java]"]
  node_model_file["Model file output"]
end

node_user(("Crawler user"))
node_web(("Web pages"))

node_user -->|"configures"| node_spider
node_spider -->|"schedules requests"| node_scheduler
node_scheduler -->|"provides requests"| node_spider
node_spider -->|"downloads request"| node_downloader
node_downloader -->|"fetches page"| node_web
node_web -->|"returns response"| node_downloader
node_downloader -->|"creates page"| node_page
node_spider -->|"processes page"| node_processor
node_processor -->|"reads page"| node_page
node_processor -->|"extracts content"| node_selectors
node_processor -->|"produces results"| node_result
node_processor -->|"submits URLs"| node_scheduler
node_user -->|"configures models"| node_oo_spider
node_oo_spider -->|"uses processor"| node_model_processor
node_model_processor -->|"extracts models"| node_model_extractor
node_model_processor -->|"reads page"| node_page
node_model_extractor -->|"selects fields"| node_selectors
node_spider -->|"dispatches results"| node_pipeline
node_pipeline -->|"accepts results"| node_result
node_model_processor -->|"stores model results"| node_result
node_model_pipeline -->|"writes models"| node_model_file
node_pipeline -->|"writes results"| node_file_pipeline
node_spider -.->|"can configure"| node_redis_scheduler
node_spider -.->|"can configure"| node_browser
node_spider -.->|"can monitor"| node_monitor
node_spider -.->|"can configure"| node_script_processor

click node_spider "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java"
click node_request "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Request.java"
click node_scheduler "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/scheduler/Scheduler.java"
click node_redis_scheduler "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/RedisScheduler.java"
click node_downloader "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java"
click node_page "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Page.java"
click node_processor "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/processor/PageProcessor.java"
click node_selectors "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/selector/Selectors.java"
click node_result "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/ResultItems.java"
click node_model_processor "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPageProcessor.java"
click node_model_extractor "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/PageModelExtractor.java"
click node_oo_spider "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/OOSpider.java"
click node_pipeline "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/Pipeline.java"
click node_model_pipeline "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPipeline.java"
click node_file_pipeline "https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/FilePipeline.java"
click node_model_file "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/pipeline/JsonFilePageModelPipeline.java"
click node_browser "https://github.com/code4craft/webmagic/blob/develop/webmagic-selenium/src/main/java/us/codecraft/webmagic/downloader/selenium/SeleniumDownloader.java"
click node_monitor "https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/monitor/SpiderMonitor.java"
click node_script_processor "https://github.com/code4craft/webmagic/blob/develop/webmagic-scripts/src/main/java/us/codecraft/webmagic/scripts/ScriptProcessor.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_spider,node_request,node_downloader,node_page,node_browser,node_monitor,node_user toneBlue
class node_processor,node_selectors,node_result,node_script_processor toneAmber
class node_model_processor,node_model_extractor,node_oo_spider toneMint
class node_scheduler,node_redis_scheduler,node_pipeline,node_model_pipeline,node_file_pipeline,node_model_file toneRose
class node_web toneIndigo
```

## Components

### Crawl runtime

- [Spider](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java): `webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java`
- [Request](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Request.java): `webmagic-core/src/main/java/us/codecraft/webmagic/Request.java`
- [HTTP downloader](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java): `webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java`
- [Page](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/Page.java): `webmagic-core/src/main/java/us/codecraft/webmagic/Page.java`
- [Browser downloaders](https://github.com/code4craft/webmagic/blob/develop/webmagic-selenium/src/main/java/us/codecraft/webmagic/downloader/selenium/SeleniumDownloader.java): `webmagic-selenium/src/main/java/us/codecraft/webmagic/downloader/selenium/SeleniumDownloader.java`
- [Spider monitor](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/monitor/SpiderMonitor.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/monitor/SpiderMonitor.java`

### Extraction

- [Page processor](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/processor/PageProcessor.java): `webmagic-core/src/main/java/us/codecraft/webmagic/processor/PageProcessor.java`
- [Content selectors](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/selector/Selectors.java): `webmagic-core/src/main/java/us/codecraft/webmagic/selector/Selectors.java`
- [Result items](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/ResultItems.java): `webmagic-core/src/main/java/us/codecraft/webmagic/ResultItems.java`
- [Script processor](https://github.com/code4craft/webmagic/blob/develop/webmagic-scripts/src/main/java/us/codecraft/webmagic/scripts/ScriptProcessor.java): `webmagic-scripts/src/main/java/us/codecraft/webmagic/scripts/ScriptProcessor.java`

### Model extensions

- [Model processor](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPageProcessor.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPageProcessor.java`
- [POJO extractor](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/PageModelExtractor.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/model/PageModelExtractor.java`
- [OOSpider](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/OOSpider.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/model/OOSpider.java`

### Scheduling and output

- [Scheduler](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/scheduler/Scheduler.java): `webmagic-core/src/main/java/us/codecraft/webmagic/scheduler/Scheduler.java`
- [Redis schedulers](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/RedisScheduler.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/RedisScheduler.java`
- [Result pipeline](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/Pipeline.java): `webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/Pipeline.java`
- [Model pipeline](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPipeline.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPipeline.java`
- [File output](https://github.com/code4craft/webmagic/blob/develop/webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/FilePipeline.java): `webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/FilePipeline.java`
- [Model file output](https://github.com/code4craft/webmagic/blob/develop/webmagic-extension/src/main/java/us/codecraft/webmagic/pipeline/JsonFilePageModelPipeline.java): `webmagic-extension/src/main/java/us/codecraft/webmagic/pipeline/JsonFilePageModelPipeline.java`

### Other components

- **Crawler user**
- **Web pages**

## Connections

- Crawler user → Spider: configures
- Spider → Scheduler: schedules requests
- Scheduler → Spider: provides requests
- Spider → HTTP downloader: downloads request
- HTTP downloader → Web pages: fetches page
- Web pages → HTTP downloader: returns response
- HTTP downloader → Page: creates page
- Spider → Page processor: processes page
- Page processor → Page: reads page
- Page processor → Content selectors: extracts content
- Page processor → Result items: produces results
- Page processor → Scheduler: submits URLs
- Crawler user → OOSpider: configures models
- OOSpider → Model processor: uses processor
- Model processor → POJO extractor: extracts models
- Model processor → Page: reads page
- POJO extractor → Content selectors: selects fields
- Spider → Result pipeline: dispatches results
- Result pipeline → Result items: accepts results
- Model processor → Result items: stores model results
- Model pipeline → Model file output: writes models
- Result pipeline → File output: writes results
- Spider → Redis schedulers: can configure
- Spider → Browser downloaders: can configure
- Spider → Spider monitor: can monitor
- Spider → Script processor: can configure