# geekan/scrapy-examples architecture

> Architecture diagram of the GitHub repository geekan/scrapy-examples, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geekan/scrapy-examples
- Repository: https://github.com/geekan/scrapy-examples

## Overview

This repository is a collection of Scrapy crawler examples and shared crawling utilities, rather than one application. The documented workflow starts a named spider, requests and traverses site pages, extracts records into items, and can use rule-based parsing; the README specifically describes a multi-depth Douban Books crawl and the reusable `parse_with_rules` approach. Examples cover directories, books, social profiles, news, forums, and other sites. Shared proxy and user-agent support is documented, but sampled material does not establish a universal item-storage flow. Relationships for unsampled spiders and settings are limited to documented or indexed evidence.

Main entry points: [`misc/spider.py`](https://github.com/geekan/scrapy-examples/blob/master/misc/spider.py) (Rule parser), [`linkedin/linkedin/linkedin/parser/HtmlParser.py`](https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/parser/HtmlParser.py) (Profile parser), [`zhihu/zhihu/pipelines.py`](https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/pipelines.py) (Zhihu pipelines), [`doubanbook/doubanbook/spiders/douban_spider.py`](https://github.com/geekan/scrapy-examples/blob/master/doubanbook/doubanbook/spiders/douban_spider.py) (Douban Books)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawl["Crawling examples"]
  node_douban["Douban Books<br/>[douban_spider.py]"]
  node_dmoz["DMOZ rules example<br/>[spider.py]"]
  node_alexa["Alexa rankings<br/>[alexa_spider.py]"]
  node_zhihu["Zhihu profiles<br/>[zhihu_spider.py]"]
  node_sis["SIS forum<br/>[sis_spider.py]"]
  node_sina["Sina news<br/>[spider.py]"]
  node_dianping["Dianping shops<br/>[spider.py]"]
end

subgraph group_shared["Shared crawling"]
  node_common["Rule parser<br/>[spider.py]"]
  node_proxy["Proxy support<br/>[proxy.py]"]
  node_agents["Agent support<br/>[agents.py]"]
end

subgraph group_linkedin["LinkedIn extraction"]
  node_linkedin_spider["LinkedIn spider<br/>[LinkedinSpider.py]"]
  node_profile_parser["Profile parser<br/>[HtmlParser.py]"]
  node_homepage_parser["Homepage parser<br/>[LinkedinParser.py]"]
end

subgraph group_outputs["Item outputs"]
  node_zhihu_items["Zhihu item<br/>[items.py]"]
  node_zhihu_pipeline["Zhihu pipelines<br/>[pipelines.py]"]
  node_alexa_items["Alexa items<br/>[items.py]"]
  node_linkedin_items["LinkedIn items<br/>[items.py]"]
  node_mysql_pipeline["Sports MySQL pipeline<br/>[pipelines.py]"]
  node_mysql_driver["MySQL driver<br/>[mysqldriver.py]"]
end

node_user(("Crawler operator"))
node_scrapy["Scrapy runtime"]
node_mysql[("MySQL database")]
node_redis[("Redis")]
node_jsonfile["JSON file"]
node_target_sites(("Target websites"))
node_baidu(("Baidu geocoder"))

node_user -->|"starts crawl"| node_scrapy
node_scrapy -->|"runs spider"| node_douban
node_scrapy -->|"runs spider"| node_dmoz
node_scrapy -->|"runs spider"| node_alexa
node_scrapy -->|"runs spider"| node_zhihu
node_scrapy -->|"runs spider"| node_linkedin_spider
node_douban -->|"crawls pages"| node_target_sites
node_dmoz -->|"uses rules"| node_common
node_sina -->|"imports parser"| node_common
node_zhihu -->|"creates items"| node_zhihu_items
node_zhihu_pipeline -->|"writes JSON"| node_jsonfile
node_zhihu_pipeline -->|"reads and writes"| node_redis
node_alexa -->|"creates items"| node_alexa_items
node_linkedin_spider -.->|"parses profiles"| node_profile_parser
node_profile_parser -->|"creates profiles"| node_linkedin_items
node_profile_parser -->|"extracts homepage"| node_homepage_parser
node_mysql_pipeline -->|"calls insert"| node_mysql_driver
node_mysql_driver -->|"executes SQL"| node_mysql
node_dianping -->|"geocodes address"| node_baidu
node_common -.->|"uses proxies"| node_proxy
node_common -.->|"uses agents"| node_agents

click node_douban "https://github.com/geekan/scrapy-examples/blob/master/doubanbook/doubanbook/spiders/douban_spider.py"
click node_dmoz "https://github.com/geekan/scrapy-examples/blob/master/dmoz/dmoz/spiders/spider.py"
click node_alexa "https://github.com/geekan/scrapy-examples/blob/master/alexa/alexa/spiders/alexa_spider.py"
click node_zhihu "https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/spiders/zhihu_spider.py"
click node_sis "https://github.com/geekan/scrapy-examples/blob/master/sis/sis/spiders/sis_spider.py"
click node_sina "https://github.com/geekan/scrapy-examples/blob/master/sinanews/sinanews/spiders/spider.py"
click node_dianping "https://github.com/geekan/scrapy-examples/blob/master/dianping/dianping/spiders/spider.py"
click node_linkedin_spider "https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/spiders/LinkedinSpider.py"
click node_common "https://github.com/geekan/scrapy-examples/blob/master/misc/spider.py"
click node_proxy "https://github.com/geekan/scrapy-examples/blob/master/misc/proxy.py"
click node_agents "https://github.com/geekan/scrapy-examples/blob/master/misc/agents.py"
click node_profile_parser "https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/parser/HtmlParser.py"
click node_homepage_parser "https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/parser/LinkedinParser.py"
click node_zhihu_items "https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/items.py"
click node_zhihu_pipeline "https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/pipelines.py"
click node_alexa_items "https://github.com/geekan/scrapy-examples/blob/master/alexa/alexa/items.py"
click node_linkedin_items "https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/items.py"
click node_mysql_pipeline "https://github.com/geekan/scrapy-examples/blob/master/zhibo8/zhibo8/pipelines.py"
click node_mysql_driver "https://github.com/geekan/scrapy-examples/blob/master/zhibo8/zhibo8/utils/mysqldriver.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_douban,node_dmoz,node_alexa,node_zhihu,node_sis,node_sina,node_dianping toneBlue
class node_common,node_proxy,node_agents,node_mysql,node_redis toneAmber
class node_linkedin_spider,node_profile_parser,node_homepage_parser toneMint
class node_zhihu_items,node_zhihu_pipeline,node_alexa_items,node_linkedin_items,node_mysql_pipeline,node_mysql_driver toneRose
class node_user,node_scrapy,node_jsonfile,node_target_sites,node_baidu toneIndigo
```

## Components

### Crawling examples

- [Douban Books](https://github.com/geekan/scrapy-examples/blob/master/doubanbook/doubanbook/spiders/douban_spider.py): `doubanbook/doubanbook/spiders/douban_spider.py`
- [DMOZ rules example](https://github.com/geekan/scrapy-examples/blob/master/dmoz/dmoz/spiders/spider.py): `dmoz/dmoz/spiders/spider.py`
- [Alexa rankings](https://github.com/geekan/scrapy-examples/blob/master/alexa/alexa/spiders/alexa_spider.py): `alexa/alexa/spiders/alexa_spider.py`
- [Zhihu profiles](https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/spiders/zhihu_spider.py): `zhihu/zhihu/spiders/zhihu_spider.py`
- [SIS forum](https://github.com/geekan/scrapy-examples/blob/master/sis/sis/spiders/sis_spider.py): `sis/sis/spiders/sis_spider.py`
- [Sina news](https://github.com/geekan/scrapy-examples/blob/master/sinanews/sinanews/spiders/spider.py): `sinanews/sinanews/spiders/spider.py`
- [Dianping shops](https://github.com/geekan/scrapy-examples/blob/master/dianping/dianping/spiders/spider.py): `dianping/dianping/spiders/spider.py`

### Shared crawling

- [Rule parser](https://github.com/geekan/scrapy-examples/blob/master/misc/spider.py): `misc/spider.py`
- [Proxy support](https://github.com/geekan/scrapy-examples/blob/master/misc/proxy.py): `misc/proxy.py`
- [Agent support](https://github.com/geekan/scrapy-examples/blob/master/misc/agents.py): `misc/agents.py`

### LinkedIn extraction

- [LinkedIn spider](https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/spiders/LinkedinSpider.py): `linkedin/linkedin/linkedin/spiders/LinkedinSpider.py`
- [Profile parser](https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/parser/HtmlParser.py): `linkedin/linkedin/linkedin/parser/HtmlParser.py`
- [Homepage parser](https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/parser/LinkedinParser.py): `linkedin/linkedin/linkedin/parser/LinkedinParser.py`

### Item outputs

- [Zhihu item](https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/items.py): `zhihu/zhihu/items.py`
- [Zhihu pipelines](https://github.com/geekan/scrapy-examples/blob/master/zhihu/zhihu/pipelines.py): `zhihu/zhihu/pipelines.py`
- [Alexa items](https://github.com/geekan/scrapy-examples/blob/master/alexa/alexa/items.py): `alexa/alexa/items.py`
- [LinkedIn items](https://github.com/geekan/scrapy-examples/blob/master/linkedin/linkedin/linkedin/items.py): `linkedin/linkedin/linkedin/items.py`
- [Sports MySQL pipeline](https://github.com/geekan/scrapy-examples/blob/master/zhibo8/zhibo8/pipelines.py): `zhibo8/zhibo8/pipelines.py`
- [MySQL driver](https://github.com/geekan/scrapy-examples/blob/master/zhibo8/zhibo8/utils/mysqldriver.py): `zhibo8/zhibo8/utils/mysqldriver.py`

### Other components

- **Crawler operator**
- **Scrapy runtime**
- **MySQL database**
- **Redis**
- **JSON file**
- **Target websites**
- **Baidu geocoder**

## Connections

- Crawler operator → Scrapy runtime: starts crawl
- Scrapy runtime → Douban Books: runs spider
- Scrapy runtime → DMOZ rules example: runs spider
- Scrapy runtime → Alexa rankings: runs spider
- Scrapy runtime → Zhihu profiles: runs spider
- Scrapy runtime → LinkedIn spider: runs spider
- Douban Books → Target websites: crawls pages
- DMOZ rules example → Rule parser: uses rules
- Sina news → Rule parser: imports parser (evidence: `sinanews/sinanews/spiders/spider.py`)
- Zhihu profiles → Zhihu item: creates items (evidence: `zhihu/zhihu/spiders/zhihu_spider.py`)
- Zhihu pipelines → JSON file: writes JSON (evidence: `zhihu/zhihu/pipelines.py`)
- Zhihu pipelines → Redis: reads and writes (evidence: `zhihu/zhihu/pipelines.py`)
- Alexa rankings → Alexa items: creates items (evidence: `alexa/alexa/spiders/alexa_spider.py`)
- LinkedIn spider → Profile parser: parses profiles
- Profile parser → LinkedIn items: creates profiles (evidence: `linkedin/linkedin/linkedin/parser/HtmlParser.py`)
- Profile parser → Homepage parser: extracts homepage (evidence: `linkedin/linkedin/linkedin/parser/HtmlParser.py`)
- Sports MySQL pipeline → MySQL driver: calls insert (evidence: `zhibo8/zhibo8/pipelines.py`)
- MySQL driver → MySQL database: executes SQL (evidence: `zhibo8/zhibo8/utils/mysqldriver.py`)
- Dianping shops → Baidu geocoder: geocodes address (evidence: `dianping/dianping/spiders/spider.py`)
- Rule parser → Proxy support: uses proxies
- Rule parser → Agent support: uses agents
