How code4craft/webmagic works
WebMagic is a Java web-crawling framework covering URL management, downloading, extraction, and persistence. A user configures a Spider with a page processor and starting URLs; the crawler schedules requests, downloads pages, extracts fields and follow-up URLs, then sends results to pipelines. Annotation-based POJO models provide an alternate extraction path. The tree and README document distributed scheduling and browser integrations; sampled source confirms pipeline behavior, but Spider’s internal dispatch is not sampled, so uncertain implementation-level links are omitted.
Main entry points: Spider.java Spider · PageProcessor.java Page processor · ModelPageProcessor.java Model processor · HttpClientDownloader.java HTTP downloader
Components 21
Crawl runtime
- Spider
webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java - Request
webmagic-core/src/main/java/us/codecraft/webmagic/Request.java - HTTP downloader
webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java - Page
webmagic-core/src/main/java/us/codecraft/webmagic/Page.java - Browser downloaders
webmagic-selenium/src/main/java/us/codecraft/webmagic/downloader/selenium/SeleniumDownloader.java - Spider monitor
webmagic-extension/src/main/java/us/codecraft/webmagic/monitor/SpiderMonitor.java
Extraction
- Page processor
webmagic-core/src/main/java/us/codecraft/webmagic/processor/PageProcessor.java - Content selectors
webmagic-core/src/main/java/us/codecraft/webmagic/selector/Selectors.java - Result items
webmagic-core/src/main/java/us/codecraft/webmagic/ResultItems.java - Script processor
webmagic-scripts/src/main/java/us/codecraft/webmagic/scripts/ScriptProcessor.java
Model extensions
- Model processor
webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPageProcessor.java - POJO extractor
webmagic-extension/src/main/java/us/codecraft/webmagic/model/PageModelExtractor.java - OOSpider
webmagic-extension/src/main/java/us/codecraft/webmagic/model/OOSpider.java
Scheduling and output
- Scheduler
webmagic-core/src/main/java/us/codecraft/webmagic/scheduler/Scheduler.java - Redis schedulers
webmagic-extension/src/main/java/us/codecraft/webmagic/scheduler/RedisScheduler.java - Result pipeline
webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/Pipeline.java - Model pipeline
webmagic-extension/src/main/java/us/codecraft/webmagic/model/ModelPipeline.java - File output
webmagic-core/src/main/java/us/codecraft/webmagic/pipeline/FilePipeline.java - Model file output
webmagic-extension/src/main/java/us/codecraft/webmagic/pipeline/JsonFilePageModelPipeline.java
Other components
- Crawler user
- Web pages
Connections 26
- Crawler user → Spider: configures
- Spider → Scheduler: schedules requests
- Scheduler → Spider: provides requests
- Spider → HTTP downloader: downloads request
- HTTP downloader → Web pages: fetches page
- Web pages → HTTP downloader: returns response
- HTTP downloader → Page: creates page
- Spider → Page processor: processes page
- Page processor → Page: reads page
- Page processor → Content selectors: extracts content
- Page processor → Result items: produces results
- Page processor → Scheduler: submits URLs
- Crawler user → OOSpider: configures models
- OOSpider → Model processor: uses processor
- Model processor → POJO extractor: extracts models
- Model processor → Page: reads page
- POJO extractor → Content selectors: selects fields
- Spider → Result pipeline: dispatches results
- Result pipeline → Result items: accepts results
- Model processor → Result items: stores model results
- Model pipeline → Model file output: writes models
- Result pipeline → File output: writes results
- Spider → Redis schedulers: can configure
- Spider → Browser downloaders: can configure
- Spider → Spider monitor: can monitor
- Spider → Script processor: can configure