# lining0806/pythonspidernotes architecture

> Architecture diagram of the GitHub repository lining0806/pythonspidernotes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lining0806/pythonspidernotes
- Repository: https://github.com/lining0806/pythonspidernotes

## Overview

This repository is a collection of crawler examples rather than one integrated application. Its common workflow is to request or automate access to a target, extract and sometimes normalize content, then save results. The graph preserves the major examples documented in the README, including browser-driven flight scraping, authenticated access, CAPTCHA OCR, Scrapy, and parallel Wallstreetcn collection. Source excerpts are incomplete, so links for unsampled implementations and configuration are limited to documented behavior or omitted where wiring is uncertain.

Main entry points: [`NewsSpider/NewsSpider.py`](https://github.com/lining0806/pythonspidernotes/blob/master/NewsSpider/NewsSpider.py) (News ranking crawler), [`WechatSearchProjects/Wechatproject/Wechatproject/spiders/spider.py`](https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Wechatproject/Wechatproject/spiders/spider.py) (Wechat Scrapy spider), [`QunarSpider/QunarSpider.py`](https://github.com/lining0806/pythonspidernotes/blob/master/QunarSpider/QunarSpider.py) (Flight scraper), [`ZhihuSpider/ZhihuSpider.py`](https://github.com/lining0806/pythonspidernotes/blob/master/ZhihuSpider/ZhihuSpider.py) (Zhihu login scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Content collection"]
  node_news["News ranking crawler<br/>[NewsSpider.py]"]
  node_wechat_spider["Wechat Scrapy spider<br/>[spider.py]"]
  node_wechat_main["Scrapy launch script<br/>[Spider_Main.py]"]
end

subgraph group_special["Browser and access"]
  node_qunar["Flight scraper<br/>[QunarSpider.py]"]
  node_zhihu["Zhihu login scraper<br/>[ZhihuSpider.py]"]
  node_captcha["Captcha OCR wrapper<br/>[pytesser_pro.py]"]
end

subgraph group_processing["Extraction and storage"]
  node_ticket_files["Saved ticket pages"]
  node_ocr_util["Image file handling<br/>[util.py]"]
  node_wechat_pipeline["Wechat item pipeline<br/>[pipelines.py]"]
  node_wechat_db[("MongoDB results")]
  node_wallstreet_db[("Live-news MongoDB")]
end

subgraph group_parallel["Parallel collectors"]
  node_wechat_multiprocess["Multiprocess Wechat scraper"]
  node_wallstreet_python["Python live-news scraper"]
  node_wallstreet_java["Java live-news scraper"]
end

node_operator(("Crawler operator"))
node_news_site(("网易新闻 site"))
node_qunar_site(("Qunar flight site"))
node_browser["Selenium browser"]
node_credentials["Login and cookies<br/>[config.ini]"]
node_zhihu_site(("Zhihu site"))
node_tesseract["Tesseract OCR"]
node_wechat_site(("Sogou Wechat search"))
node_live_api(("Wallstreetcn live API"))

node_operator -->|"starts"| node_news
node_news -->|"requests pages"| node_news_site
node_news -->|"extracts and saves"| node_news
node_operator -->|"starts"| node_qunar
node_qunar -->|"controls"| node_browser
node_browser -->|"loads pages"| node_qunar_site
node_qunar -->|"writes HTML"| node_ticket_files
node_operator -->|"starts"| node_zhihu
node_zhihu -->|"reads"| node_credentials
node_zhihu -->|"logs in and fetches"| node_zhihu_site
node_captcha -->|"stages image"| node_ocr_util
node_captcha -->|"runs OCR"| node_tesseract
node_operator -->|"starts"| node_wechat_main
node_wechat_main -->|"launches crawl"| node_wechat_spider
node_wechat_spider -->|"requests search and articles"| node_wechat_site
node_wechat_spider -->|"yields items"| node_wechat_pipeline
node_wechat_pipeline -->|"inserts records"| node_wechat_db
node_operator -->|"starts"| node_wechat_multiprocess
node_wechat_multiprocess -->|"requests results and articles"| node_wechat_site
node_wechat_multiprocess -->|"saves records"| node_wechat_db
node_operator -->|"starts"| node_wallstreet_python
node_wallstreet_python -->|"requests pages"| node_live_api
node_wallstreet_python -->|"saves categorized records"| node_wallstreet_db
node_operator -->|"starts"| node_wallstreet_java
node_wallstreet_java -->|"requests pages"| node_live_api
node_wallstreet_java -->|"inserts records"| node_wallstreet_db

click node_news "https://github.com/lining0806/pythonspidernotes/blob/master/NewsSpider/NewsSpider.py"
click node_qunar "https://github.com/lining0806/pythonspidernotes/blob/master/QunarSpider/QunarSpider.py"
click node_zhihu "https://github.com/lining0806/pythonspidernotes/blob/master/ZhihuSpider/ZhihuSpider.py"
click node_credentials "https://github.com/lining0806/pythonspidernotes/blob/master/ZhihuSpider/config.ini"
click node_captcha "https://github.com/lining0806/pythonspidernotes/blob/master/Captcha1/pytesser_pro/pytesser_pro.py"
click node_ocr_util "https://github.com/lining0806/pythonspidernotes/blob/master/Captcha1/pytesser_pro/util.py"
click node_wechat_spider "https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Wechatproject/Wechatproject/spiders/spider.py"
click node_wechat_main "https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Spider_Main.py"
click node_wechat_pipeline "https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Wechatproject/Wechatproject/pipelines.py"
click node_wechat_multiprocess "https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/WechatSearchTest.py"
click node_wallstreet_python "https://github.com/lining0806/pythonspidernotes/blob/master/Spider_Python/WallstreetcnSaveTest.py"
click node_wallstreet_java "https://github.com/lining0806/pythonspidernotes/blob/master/Spider_Java/Spider_Java1/src/wallstreetcnsave/WallstreetcnSaveTest.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_news,node_wechat_spider,node_wechat_main,node_browser toneBlue
class node_qunar,node_zhihu,node_captcha toneAmber
class node_ticket_files,node_ocr_util,node_wechat_pipeline,node_wechat_db,node_wallstreet_db,node_live_api toneMint
class node_wechat_multiprocess,node_wallstreet_python,node_wallstreet_java toneRose
class node_operator,node_news_site,node_qunar_site,node_zhihu_site,node_tesseract,node_wechat_site toneIndigo
class node_credentials toneTeal
```

## Components

### Content collection

- [News ranking crawler](https://github.com/lining0806/pythonspidernotes/blob/master/NewsSpider/NewsSpider.py): `NewsSpider/NewsSpider.py`
- [Wechat Scrapy spider](https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Wechatproject/Wechatproject/spiders/spider.py): `WechatSearchProjects/Wechatproject/Wechatproject/spiders/spider.py`
- [Scrapy launch script](https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Spider_Main.py): `WechatSearchProjects/Spider_Main.py`

### Browser and access

- [Flight scraper](https://github.com/lining0806/pythonspidernotes/blob/master/QunarSpider/QunarSpider.py): `QunarSpider/QunarSpider.py`
- [Zhihu login scraper](https://github.com/lining0806/pythonspidernotes/blob/master/ZhihuSpider/ZhihuSpider.py): `ZhihuSpider/ZhihuSpider.py`
- [Captcha OCR wrapper](https://github.com/lining0806/pythonspidernotes/blob/master/Captcha1/pytesser_pro/pytesser_pro.py): `Captcha1/pytesser_pro/pytesser_pro.py`

### Extraction and storage

- **Saved ticket pages**
- [Image file handling](https://github.com/lining0806/pythonspidernotes/blob/master/Captcha1/pytesser_pro/util.py): `Captcha1/pytesser_pro/util.py`
- [Wechat item pipeline](https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/Wechatproject/Wechatproject/pipelines.py): `WechatSearchProjects/Wechatproject/Wechatproject/pipelines.py`
- **MongoDB results**
- **Live-news MongoDB**

### Parallel collectors

- [Multiprocess Wechat scraper](https://github.com/lining0806/pythonspidernotes/blob/master/WechatSearchProjects/WechatSearchTest.py): `WechatSearchProjects/WechatSearchTest.py`
- [Python live-news scraper](https://github.com/lining0806/pythonspidernotes/blob/master/Spider_Python/WallstreetcnSaveTest.py): `Spider_Python/WallstreetcnSaveTest.py`
- [Java live-news scraper](https://github.com/lining0806/pythonspidernotes/blob/master/Spider_Java/Spider_Java1/src/wallstreetcnsave/WallstreetcnSaveTest.java): `Spider_Java/Spider_Java1/src/wallstreetcnsave/WallstreetcnSaveTest.java`

### Other components

- **Crawler operator**
- **网易新闻 site**
- **Qunar flight site**
- **Selenium browser**
- [Login and cookies](https://github.com/lining0806/pythonspidernotes/blob/master/ZhihuSpider/config.ini): `ZhihuSpider/config.ini`
- **Zhihu site**
- **Tesseract OCR**
- **Sogou Wechat search**
- **Wallstreetcn live API**

## Connections

- Crawler operator → News ranking crawler: starts
- News ranking crawler → 网易新闻 site: requests pages
- News ranking crawler → News ranking crawler: extracts and saves
- Crawler operator → Flight scraper: starts
- Flight scraper → Selenium browser: controls
- Selenium browser → Qunar flight site: loads pages
- Flight scraper → Saved ticket pages: writes HTML
- Crawler operator → Zhihu login scraper: starts
- Zhihu login scraper → Login and cookies: reads
- Zhihu login scraper → Zhihu site: logs in and fetches
- Captcha OCR wrapper → Image file handling: stages image
- Captcha OCR wrapper → Tesseract OCR: runs OCR
- Crawler operator → Scrapy launch script: starts
- Scrapy launch script → Wechat Scrapy spider: launches crawl
- Wechat Scrapy spider → Sogou Wechat search: requests search and articles
- Wechat Scrapy spider → Wechat item pipeline: yields items
- Wechat item pipeline → MongoDB results: inserts records
- Crawler operator → Multiprocess Wechat scraper: starts
- Multiprocess Wechat scraper → Sogou Wechat search: requests results and articles
- Multiprocess Wechat scraper → MongoDB results: saves records
- Crawler operator → Python live-news scraper: starts
- Python live-news scraper → Wallstreetcn live API: requests pages
- Python live-news scraper → Live-news MongoDB: saves categorized records
- Crawler operator → Java live-news scraper: starts
- Java live-news scraper → Wallstreetcn live API: requests pages
- Java live-news scraper → Live-news MongoDB: inserts records