How wistbean/learn_python3_spider works
This repository is a collection of Python scraping tutorials and runnable examples, not one integrated application. Its clearest end-to-end workflows are the two Scrapy projects: spiders extract records into item schemas, and pipelines write them to MongoDB. Other branches demonstrate standalone site scraping, image search/download, captcha handling, WeChat automation, and packaged gaokao score data. The graph preserves these documented capabilities; several standalone scripts and settings are represented by their tree paths, but their internals were not inspected.
Main entry points: qiushibaike_spider.py Qiushi spider · stackoverflow-python-spider.py Stack Overflow spider · fuck_bilibili_captcha.py Slider captcha · wechat_moment.py Moments scraper
Components 20
Scrapy crawlers
- Qiushi spider
qiushibaike/qiushibaike/spiders/qiushibaike_spider.py - Qiushi item
qiushibaike/qiushibaike/items.py - Qiushi pipeline
qiushibaike/qiushibaike/pipelines.py - Stack Overflow spider
stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py - Question item
stackoverflow/stackoverflow/items.py - Question pipeline
stackoverflow/stackoverflow/pipelines.py
Standalone scraping
- Book scrapers
dangdang_top_500.py - Douban scrapers
douban_top_250_books.py - Meme downloader
biaoqingbao/biaoqingbao.py - Meme search
biaoqingbao/search.py - Slider captcha
fuck_bilibili_captcha.py
Messaging automation
- Moments scraper
wechat_moment.py - Account scraper
wechat_public_account.py
Collected data
- Gaokao score pages
GaoKao_Score
Other components
- Tutorial reader
- MongoDB
- Qiushibaike
- Stack Overflow
- Bilibili login
Connections 16
- Qiushi spider → Qiushi item: creates items
qiushibaike/qiushibaike/spiders/qiushibaike_spider.py - Qiushi pipeline → MongoDB: writes records
qiushibaike/qiushibaike/pipelines.py - Stack Overflow spider → Question item: creates items
stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py - Question pipeline → MongoDB: writes records
stackoverflow/stackoverflow/pipelines.py - Tutorial reader → Book scrapers: runs examples
README.md - Tutorial reader → Douban scrapers: runs examples
README.md - Tutorial reader → Meme downloader: runs example
README.md - Tutorial reader → Meme search: runs example
README.md - Tutorial reader → Slider captcha: runs example
README.md - Tutorial reader → Moments scraper: runs example
README.md - Tutorial reader → Account scraper: runs example
README.md - Tutorial reader → Gaokao score pages: explores data
README.md - Qiushi spider → Qiushibaike: requests pages
qiushibaike/qiushibaike/spiders/qiushibaike_spider.py - Stack Overflow spider → Stack Overflow: requests pages
stackoverflow/stackoverflow/spiders/stackoverflow-python-spider.py - Slider captcha → Bilibili login: automates login
fuck_bilibili_captcha.py - Moments scraper → WeChat: automates browsing
wechat_moment.py