How kaifagou/scrapyspider works
This repository is a legacy Facebook-ad scraping and data-sync application. Its user-facing workflow schedules homepage collection and chains ad-source, homepage, and video-detail handlers; other spiders cover video metadata, ad pages, and media downloads. Scrapy items and pipelines represent and persist collected data, with Redis used for queues and results. A separate Manager listens for Redis refresh commands and coordinates spider notifications and MySQL-facing components. The README is minimal, and several spider and persistence implementations were not sampled, so their internal relationships are omitted or marked unverified.
Main entry points: Manager.py Refresh manager · homepage.py Homepage scheduler · RedisHelper.py Redis queues · GetVideoSpider.py Video detail handler
Components 29
Scraping workflows
- Homepage scheduler
homepage.py - Adspy spider
startSpider/AdspySpider.py - AdSector spider
startSpider/AdSectorSpider.py - Homepage spider
startSpider/HomepageSpider.py - Video detail handler
startSpider/GetVideoSpider.py - Video spiders
scrapyuniversal/spiders/VideoSpider.py - Video info spider
scrapyuniversal/spiders/VideoInfoSpider.py - Media download spider
scrapyuniversal/spiders/SaveVideoSpider.py - Ad collection spiders
scrapyuniversal/spiders/AdSectorSpider.py
Items and queues
- Homepage item
scrapyuniversal/items/Fb_Homepage_Item.py - Video item
scrapyuniversal/items/Fb_Video_Item.py - Video info item
scrapyuniversal/items/Fb_Video_Info_Item.py - Download item
scrapyuniversal/items/Fb_Download_Item.py - Redis queues
RedisHelper/RedisHelper.py - Media save pipeline
scrapyuniversal/pipelines/SaveVideoPipeline.py - Video pipelines
scrapyuniversal/pipelines/VideoRedisPipeline.py - Info pipelines
scrapyuniversal/pipelines/VideoInfoRedisPipeline.py
Sync and storage
- Refresh manager
Engine/Manager.py - Observer dispatcher
Engine/SpiderServer.py - MySQL connection
Engine/MySqlHelper.py - Homepage persistence
Engine/HomepageToMysql.py - Video persistence
Engine/VideoToMysql.py - Video updates
Engine/UpdateVideoToMysql.py - Media content sync
Engine/VideoContentFromMysql.py - Video info sync
Engine/VideoInfoFromMysql.py - Ads sync
Engine/AdsFromMysql.py
Other components
- Operator
- MySQL database
Connections 36
- Operator → Homepage scheduler: starts
- Homepage scheduler → Adspy spider: chains to
homepage.py - Homepage scheduler → AdSector spider: chains to
homepage.py - Homepage scheduler → Homepage spider: chains to
homepage.py - Homepage scheduler → Video detail handler: chains to
homepage.py - Homepage scheduler → Redis queues: connects
homepage.py - Adspy spider → Facebook: scrapes
- AdSector spider → Facebook: scrapes
- Homepage spider → Facebook: scrapes
- Video spiders → Facebook: scrapes
- Video info spider → Facebook: scrapes
- Media download spider → Facebook: downloads from
- Video detail handler → Video item: creates
- Video detail handler → Video info item: uses
- Video detail handler → Redis queues: uses
- Media save pipeline → Redis queues: queues results
scrapyuniversal/pipelines/SaveVideoPipeline.py - Video spiders → Video item: uses
- Video spiders → Video info item: uses
- Video pipelines → Redis queues: persists items
- Info pipelines → Redis queues: persists items
- Refresh manager → Redis queues: polls commands
Engine/Manager.py - Refresh manager → Observer dispatcher: dispatches
Engine/Manager.py - Refresh manager → Homepage persistence: registers observer
Engine/Manager.py - Refresh manager → Video updates: registers observer
Engine/Manager.py - Refresh manager → Media content sync: registers observer
Engine/Manager.py - Refresh manager → Video info sync: registers observer
Engine/Manager.py - Refresh manager → Ads sync: registers observer
Engine/Manager.py - Refresh manager → Video persistence: constructs
Engine/Manager.py - Refresh manager → Video info sync: constructs
Engine/Manager.py - Homepage persistence → MySQL connection: uses
Engine/HomepageToMysql.py - Homepage persistence → MySQL database: writes
- Video persistence → MySQL database: writes
- Video updates → MySQL database: updates
- Media content sync → MySQL database: reads
- Video info sync → MySQL database: reads
- Ads sync → MySQL database: reads