How lining0806/pythonspidernotes works
This repository is a collection of crawler examples rather than one integrated application. Its common workflow is to request or automate access to a target, extract and sometimes normalize content, then save results. The graph preserves the major examples documented in the README, including browser-driven flight scraping, authenticated access, CAPTCHA OCR, Scrapy, and parallel Wallstreetcn collection. Source excerpts are incomplete, so links for unsampled implementations and configuration are limited to documented behavior or omitted where wiring is uncertain.
Main entry points: NewsSpider.py News ranking crawler · spider.py Wechat Scrapy spider · QunarSpider.py Flight scraper · ZhihuSpider.py Zhihu login scraper
Components 23
Content collection
- News ranking crawler
NewsSpider/NewsSpider.py - Wechat Scrapy spider
WechatSearchProjects/Wechatproject/Wechatproject/spiders/spider.py - Scrapy launch script
WechatSearchProjects/Spider_Main.py
Browser and access
- Flight scraper
QunarSpider/QunarSpider.py - Zhihu login scraper
ZhihuSpider/ZhihuSpider.py - Captcha OCR wrapper
Captcha1/pytesser_pro/pytesser_pro.py
Extraction and storage
- Saved ticket pages
- Image file handling
Captcha1/pytesser_pro/util.py - Wechat item pipeline
WechatSearchProjects/Wechatproject/Wechatproject/pipelines.py - MongoDB results
- Live-news MongoDB
Parallel collectors
- Multiprocess Wechat scraper
WechatSearchProjects/WechatSearchTest.py - Python live-news scraper
Spider_Python/WallstreetcnSaveTest.py - Java live-news scraper
Spider_Java/Spider_Java1/src/wallstreetcnsave/WallstreetcnSaveTest.java
Other components
- Crawler operator
- 网易新闻 site
- Qunar flight site
- Selenium browser
- Login and cookies
ZhihuSpider/config.ini - Zhihu site
- Tesseract OCR
- Sogou Wechat search
- Wallstreetcn live API
Connections 26
- Crawler operator → News ranking crawler: starts
- News ranking crawler → 网易新闻 site: requests pages
- News ranking crawler → News ranking crawler: extracts and saves
- Crawler operator → Flight scraper: starts
- Flight scraper → Selenium browser: controls
- Selenium browser → Qunar flight site: loads pages
- Flight scraper → Saved ticket pages: writes HTML
- Crawler operator → Zhihu login scraper: starts
- Zhihu login scraper → Login and cookies: reads
- Zhihu login scraper → Zhihu site: logs in and fetches
- Captcha OCR wrapper → Image file handling: stages image
- Captcha OCR wrapper → Tesseract OCR: runs OCR
- Crawler operator → Scrapy launch script: starts
- Scrapy launch script → Wechat Scrapy spider: launches crawl
- Wechat Scrapy spider → Sogou Wechat search: requests search and articles
- Wechat Scrapy spider → Wechat item pipeline: yields items
- Wechat item pipeline → MongoDB results: inserts records
- Crawler operator → Multiprocess Wechat scraper: starts
- Multiprocess Wechat scraper → Sogou Wechat search: requests results and articles
- Multiprocess Wechat scraper → MongoDB results: saves records
- Crawler operator → Python live-news scraper: starts
- Python live-news scraper → Wallstreetcn live API: requests pages
- Python live-news scraper → Live-news MongoDB: saves categorized records
- Crawler operator → Java live-news scraper: starts
- Java live-news scraper → Wallstreetcn live API: requests pages
- Java live-news scraper → Live-news MongoDB: inserts records