How geekan/scrapy-examples works
This repository is a collection of Scrapy crawler examples and shared crawling utilities, rather than one application. The documented workflow starts a named spider, requests and traverses site pages, extracts records into items, and can use rule-based parsing; the README specifically describes a multi-depth Douban Books crawl and the reusable parse_with_rules approach. Examples cover directories, books, social profiles, news, forums, and other sites. Shared proxy and user-agent support is documented, but sampled material does not establish a universal item-storage flow. Relationships for unsampled spiders and settings are limited to documented or indexed evidence.
Main entry points: spider.py Rule parser · HtmlParser.py Profile parser · pipelines.py Zhihu pipelines · douban_spider.py Douban Books
Components 26
Crawling examples
- Douban Books
doubanbook/doubanbook/spiders/douban_spider.py - DMOZ rules example
dmoz/dmoz/spiders/spider.py - Alexa rankings
alexa/alexa/spiders/alexa_spider.py - Zhihu profiles
zhihu/zhihu/spiders/zhihu_spider.py - SIS forum
sis/sis/spiders/sis_spider.py - Sina news
sinanews/sinanews/spiders/spider.py - Dianping shops
dianping/dianping/spiders/spider.py
Shared crawling
- Rule parser
misc/spider.py - Proxy support
misc/proxy.py - Agent support
misc/agents.py
LinkedIn extraction
- LinkedIn spider
linkedin/linkedin/linkedin/spiders/LinkedinSpider.py - Profile parser
linkedin/linkedin/linkedin/parser/HtmlParser.py - Homepage parser
linkedin/linkedin/linkedin/parser/LinkedinParser.py
Item outputs
- Zhihu item
zhihu/zhihu/items.py - Zhihu pipelines
zhihu/zhihu/pipelines.py - Alexa items
alexa/alexa/items.py - LinkedIn items
linkedin/linkedin/linkedin/items.py - Sports MySQL pipeline
zhibo8/zhibo8/pipelines.py - MySQL driver
zhibo8/zhibo8/utils/mysqldriver.py
Other components
- Crawler operator
- Scrapy runtime
- MySQL database
- Redis
- JSON file
- Target websites
- Baidu geocoder
Connections 21
- Crawler operator → Scrapy runtime: starts crawl
- Scrapy runtime → Douban Books: runs spider
- Scrapy runtime → DMOZ rules example: runs spider
- Scrapy runtime → Alexa rankings: runs spider
- Scrapy runtime → Zhihu profiles: runs spider
- Scrapy runtime → LinkedIn spider: runs spider
- Douban Books → Target websites: crawls pages
- DMOZ rules example → Rule parser: uses rules
- Sina news → Rule parser: imports parser
sinanews/sinanews/spiders/spider.py - Zhihu profiles → Zhihu item: creates items
zhihu/zhihu/spiders/zhihu_spider.py - Zhihu pipelines → JSON file: writes JSON
zhihu/zhihu/pipelines.py - Zhihu pipelines → Redis: reads and writes
zhihu/zhihu/pipelines.py - Alexa rankings → Alexa items: creates items
alexa/alexa/spiders/alexa_spider.py - LinkedIn spider → Profile parser: parses profiles
- Profile parser → LinkedIn items: creates profiles
linkedin/linkedin/linkedin/parser/HtmlParser.py - Profile parser → Homepage parser: extracts homepage
linkedin/linkedin/linkedin/parser/HtmlParser.py - Sports MySQL pipeline → MySQL driver: calls insert
zhibo8/zhibo8/pipelines.py - MySQL driver → MySQL database: executes SQL
zhibo8/zhibo8/utils/mysqldriver.py - Dianping shops → Baidu geocoder: geocodes address
dianping/dianping/spiders/spider.py - Rule parser → Proxy support: uses proxies
- Rule parser → Agent support: uses agents