# wistbean/learn_python3_spider architecture

> Architecture diagram of the GitHub repository wistbean/learn_python3_spider, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wistbean/learn_python3_spider
- Repository: https://github.com/wistbean/learn_python3_spider

## Overview

This repository is a collection of Python scraping tutorials and runnable examples, not one integrated application. Its clearest end-to-end workflows are the two Scrapy projects: spiders extract records into item schemas, and pipelines write them to MongoDB. Other branches demonstrate standalone site scraping, image search/download, captcha handling, WeChat automation, and packaged gaokao score data. The graph preserves these documented capabilities; several standalone scripts and settings are represented by their tree paths, but their internals were not inspected.

Main entry points: [`qiushibaike/qiushibaike/spiders/qiushibaike_spider.py`](https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/spiders/qiushibaike_spider.py) (Qiushi spider), [`stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py`](https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py) (Stack Overflow spider), [`fuck_bilibili_captcha.py`](https://github.com/wistbean/learn_python3_spider/blob/master/fuck_bilibili_captcha.py) (Slider captcha), [`wechat_moment.py`](https://github.com/wistbean/learn_python3_spider/blob/master/wechat_moment.py) (Moments scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scrapy["Scrapy crawlers"]
  node_qiushi_spider["Qiushi spider"]
  node_qiushi_item["Qiushi item<br/>[items.py]"]
  node_qiushi_pipeline["Qiushi pipeline<br/>[pipelines.py]"]
  node_stackoverflow_spider["Stack Overflow spider"]
  node_stackoverflow_item["Question item<br/>[items.py]"]
  node_stackoverflow_pipeline["Question pipeline<br/>[pipelines.py]"]
end

subgraph group_standalone["Standalone scraping"]
  node_book_scrapers["Book scrapers"]
  node_douban_scrapers["Douban scrapers"]
  node_image_downloader["Meme downloader<br/>[biaoqingbao.py]"]
  node_image_search["Meme search<br/>[search.py]"]
  node_captcha_solver["Slider captcha"]
end

subgraph group_automation["Messaging automation"]
  node_moment_scraper["Moments scraper<br/>[wechat_moment.py]"]
  node_account_scraper["Account scraper"]
end

subgraph group_datasets["Collected data"]
  node_score_data["Gaokao score pages"]
end

node_reader(("Tutorial reader"))
node_mongo[("MongoDB")]
node_qiushi_site["Qiushibaike"]
node_stackoverflow_site["Stack Overflow"]
node_bilibili["Bilibili login"]
node_wechat["WeChat"]

node_qiushi_spider -->|"creates items"| node_qiushi_item
node_qiushi_pipeline -->|"writes records"| node_mongo
node_stackoverflow_spider -->|"creates items"| node_stackoverflow_item
node_stackoverflow_pipeline -->|"writes records"| node_mongo
node_reader -->|"runs examples"| node_book_scrapers
node_reader -->|"runs examples"| node_douban_scrapers
node_reader -->|"runs example"| node_image_downloader
node_reader -->|"runs example"| node_image_search
node_reader -->|"runs example"| node_captcha_solver
node_reader -->|"runs example"| node_moment_scraper
node_reader -->|"runs example"| node_account_scraper
node_reader -->|"explores data"| node_score_data
node_qiushi_spider -->|"requests pages"| node_qiushi_site
node_stackoverflow_spider -->|"requests pages"| node_stackoverflow_site
node_captcha_solver -->|"automates login"| node_bilibili
node_moment_scraper -->|"automates browsing"| node_wechat

click node_qiushi_spider "https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/spiders/qiushibaike_spider.py"
click node_qiushi_item "https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/items.py"
click node_qiushi_pipeline "https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/pipelines.py"
click node_stackoverflow_spider "https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py"
click node_stackoverflow_item "https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/items.py"
click node_stackoverflow_pipeline "https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/pipelines.py"
click node_book_scrapers "https://github.com/wistbean/learn_python3_spider/blob/master/dangdang_top_500.py"
click node_douban_scrapers "https://github.com/wistbean/learn_python3_spider/blob/master/douban_top_250_books.py"
click node_image_downloader "https://github.com/wistbean/learn_python3_spider/blob/master/biaoqingbao/biaoqingbao.py"
click node_image_search "https://github.com/wistbean/learn_python3_spider/blob/master/biaoqingbao/search.py"
click node_captcha_solver "https://github.com/wistbean/learn_python3_spider/blob/master/fuck_bilibili_captcha.py"
click node_moment_scraper "https://github.com/wistbean/learn_python3_spider/blob/master/wechat_moment.py"
click node_account_scraper "https://github.com/wistbean/learn_python3_spider/blob/master/wechat_public_account.py"
click node_score_data "https://github.com/wistbean/learn_python3_spider/tree/master/GaoKao_Score"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_qiushi_spider,node_qiushi_item,node_qiushi_pipeline,node_stackoverflow_spider,node_stackoverflow_item,node_stackoverflow_pipeline toneBlue
class node_book_scrapers,node_douban_scrapers,node_image_downloader,node_image_search,node_captcha_solver,node_mongo toneAmber
class node_moment_scraper,node_account_scraper toneMint
class node_score_data toneRose
class node_reader,node_qiushi_site,node_stackoverflow_site,node_bilibili,node_wechat toneIndigo
```

## Components

### Scrapy crawlers

- [Qiushi spider](https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/spiders/qiushibaike_spider.py): `qiushibaike/qiushibaike/spiders/qiushibaike_spider.py`
- [Qiushi item](https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/items.py): `qiushibaike/qiushibaike/items.py`
- [Qiushi pipeline](https://github.com/wistbean/learn_python3_spider/blob/master/qiushibaike/qiushibaike/pipelines.py): `qiushibaike/qiushibaike/pipelines.py`
- [Stack Overflow spider](https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py): `stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py`
- [Question item](https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/items.py): `stackoverflow/stackoverflow/items.py`
- [Question pipeline](https://github.com/wistbean/learn_python3_spider/blob/master/stackoverflow/stackoverflow/pipelines.py): `stackoverflow/stackoverflow/pipelines.py`

### Standalone scraping

- [Book scrapers](https://github.com/wistbean/learn_python3_spider/blob/master/dangdang_top_500.py): `dangdang_top_500.py`
- [Douban scrapers](https://github.com/wistbean/learn_python3_spider/blob/master/douban_top_250_books.py): `douban_top_250_books.py`
- [Meme downloader](https://github.com/wistbean/learn_python3_spider/blob/master/biaoqingbao/biaoqingbao.py): `biaoqingbao/biaoqingbao.py`
- [Meme search](https://github.com/wistbean/learn_python3_spider/blob/master/biaoqingbao/search.py): `biaoqingbao/search.py`
- [Slider captcha](https://github.com/wistbean/learn_python3_spider/blob/master/fuck_bilibili_captcha.py): `fuck_bilibili_captcha.py`

### Messaging automation

- [Moments scraper](https://github.com/wistbean/learn_python3_spider/blob/master/wechat_moment.py): `wechat_moment.py`
- [Account scraper](https://github.com/wistbean/learn_python3_spider/blob/master/wechat_public_account.py): `wechat_public_account.py`

### Collected data

- [Gaokao score pages](https://github.com/wistbean/learn_python3_spider/tree/master/GaoKao_Score): `GaoKao_Score`

### Other components

- **Tutorial reader**
- **MongoDB**
- **Qiushibaike**
- **Stack Overflow**
- **Bilibili login**
- **WeChat**

## Connections

- Qiushi spider → Qiushi item: creates items (evidence: `qiushibaike/qiushibaike/spiders/qiushibaike_spider.py`)
- Qiushi pipeline → MongoDB: writes records (evidence: `qiushibaike/qiushibaike/pipelines.py`)
- Stack Overflow spider → Question item: creates items (evidence: `stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py`)
- Question pipeline → MongoDB: writes records (evidence: `stackoverflow/stackoverflow/pipelines.py`)
- Tutorial reader → Book scrapers: runs examples (evidence: `README.md`)
- Tutorial reader → Douban scrapers: runs examples (evidence: `README.md`)
- Tutorial reader → Meme downloader: runs example (evidence: `README.md`)
- Tutorial reader → Meme search: runs example (evidence: `README.md`)
- Tutorial reader → Slider captcha: runs example (evidence: `README.md`)
- Tutorial reader → Moments scraper: runs example (evidence: `README.md`)
- Tutorial reader → Account scraper: runs example (evidence: `README.md`)
- Tutorial reader → Gaokao score pages: explores data (evidence: `README.md`)
- Qiushi spider → Qiushibaike: requests pages (evidence: `qiushibaike/qiushibaike/spiders/qiushibaike_spider.py`)
- Stack Overflow spider → Stack Overflow: requests pages (evidence: `stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py`)
- Slider captcha → Bilibili login: automates login (evidence: `fuck_bilibili_captcha.py`)
- Moments scraper → WeChat: automates browsing (evidence: `wechat_moment.py`)
