# kaifagou/scrapyspider architecture

> Architecture diagram of the GitHub repository kaifagou/scrapyspider, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaifagou/scrapyspider
- Repository: https://github.com/kaifagou/scrapyspider

## Overview

This repository is a legacy Facebook-ad scraping and data-sync application. Its user-facing workflow schedules homepage collection and chains ad-source, homepage, and video-detail handlers; other spiders cover video metadata, ad pages, and media downloads. Scrapy items and pipelines represent and persist collected data, with Redis used for queues and results. A separate Manager listens for Redis refresh commands and coordinates spider notifications and MySQL-facing components. The README is minimal, and several spider and persistence implementations were not sampled, so their internal relationships are omitted or marked unverified.

Main entry points: [`Engine/Manager.py`](https://github.com/kaifagou/scrapyspider/blob/master/Engine/Manager.py) (Refresh manager), [`homepage.py`](https://github.com/kaifagou/scrapyspider/blob/master/homepage.py) (Homepage scheduler), [`RedisHelper/RedisHelper.py`](https://github.com/kaifagou/scrapyspider/blob/master/RedisHelper/RedisHelper.py) (Redis queues), [`startSpider/GetVideoSpider.py`](https://github.com/kaifagou/scrapyspider/blob/master/startSpider/GetVideoSpider.py) (Video detail handler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scrape["Scraping workflows"]
  node_scheduler["Homepage scheduler<br/>[homepage.py]"]
  node_adspy["Adspy spider<br/>[AdspySpider.py]"]
  node_adsector["AdSector spider<br/>[AdSectorSpider.py]"]
  node_homepage_spider["Homepage spider<br/>[HomepageSpider.py]"]
  node_get_video_handler["Video detail handler<br/>[GetVideoSpider.py]"]
  node_video_spiders["Video spiders<br/>[VideoSpider.py]"]
  node_video_info_spider["Video info spider<br/>[VideoInfoSpider.py]"]
  node_download_spider["Media download spider<br/>[SaveVideoSpider.py]"]
  node_ad_spiders["Ad collection spiders<br/>[AdSectorSpider.py]"]
end

subgraph group_records["Items and queues"]
  node_homepage_data["Homepage item"]
  node_video_item["Video item<br/>[Fb_Video_Item.py]"]
  node_video_info_item["Video info item"]
  node_download_item["Download item"]
  node_redis[("Redis queues<br/>[RedisHelper.py]")]
  node_save_pipeline["Media save pipeline"]
  node_video_pipelines["Video pipelines"]
  node_info_pipelines["Info pipelines"]
end

subgraph group_sync["Sync and storage"]
  node_manager["Refresh manager<br/>[Manager.py]"]
  node_spider_server["Observer dispatcher<br/>[SpiderServer.py]"]
  node_mysql_helpers["MySQL connection<br/>[MySqlHelper.py]"]
  node_homepage_mysql["Homepage persistence<br/>[HomepageToMysql.py]"]
  node_video_mysql["Video persistence<br/>[VideoToMysql.py]"]
  node_video_update["Video updates"]
  node_content_mysql["Media content sync"]
  node_info_mysql["Video info sync"]
  node_ads_mysql["Ads sync<br/>[AdsFromMysql.py]"]
end

node_operator(("Operator"))
node_external_fb(("Facebook"))
node_mysql[("MySQL database")]

node_operator -.->|"starts"| node_scheduler
node_scheduler -->|"chains to"| node_adspy
node_scheduler -->|"chains to"| node_adsector
node_scheduler -->|"chains to"| node_homepage_spider
node_scheduler -->|"chains to"| node_get_video_handler
node_scheduler -->|"connects"| node_redis
node_adspy -.->|"scrapes"| node_external_fb
node_adsector -.->|"scrapes"| node_external_fb
node_homepage_spider -.->|"scrapes"| node_external_fb
node_video_spiders -.->|"scrapes"| node_external_fb
node_video_info_spider -.->|"scrapes"| node_external_fb
node_download_spider -.->|"downloads from"| node_external_fb
node_get_video_handler -->|"creates"| node_video_item
node_get_video_handler -->|"uses"| node_video_info_item
node_get_video_handler -->|"uses"| node_redis
node_save_pipeline -->|"queues results"| node_redis
node_video_spiders -->|"uses"| node_video_item
node_video_spiders -->|"uses"| node_video_info_item
node_video_pipelines -.->|"persists items"| node_redis
node_info_pipelines -.->|"persists items"| node_redis
node_manager -->|"polls commands"| node_redis
node_manager -->|"dispatches"| node_spider_server
node_manager -->|"registers observer"| node_homepage_mysql
node_manager -->|"registers observer"| node_video_update
node_manager -->|"registers observer"| node_content_mysql
node_manager -->|"registers observer"| node_info_mysql
node_manager -->|"registers observer"| node_ads_mysql
node_manager -->|"constructs"| node_video_mysql
node_manager -->|"constructs"| node_info_mysql
node_homepage_mysql -->|"uses"| node_mysql_helpers
node_homepage_mysql -.->|"writes"| node_mysql
node_video_mysql -.->|"writes"| node_mysql
node_video_update -.->|"updates"| node_mysql
node_content_mysql -.->|"reads"| node_mysql
node_info_mysql -.->|"reads"| node_mysql
node_ads_mysql -.->|"reads"| node_mysql

click node_scheduler "https://github.com/kaifagou/scrapyspider/blob/master/homepage.py"
click node_adspy "https://github.com/kaifagou/scrapyspider/blob/master/startSpider/AdspySpider.py"
click node_adsector "https://github.com/kaifagou/scrapyspider/blob/master/startSpider/AdSectorSpider.py"
click node_homepage_spider "https://github.com/kaifagou/scrapyspider/blob/master/startSpider/HomepageSpider.py"
click node_get_video_handler "https://github.com/kaifagou/scrapyspider/blob/master/startSpider/GetVideoSpider.py"
click node_video_spiders "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/VideoSpider.py"
click node_video_info_spider "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/VideoInfoSpider.py"
click node_download_spider "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/SaveVideoSpider.py"
click node_ad_spiders "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/AdSectorSpider.py"
click node_homepage_data "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Homepage_Item.py"
click node_video_item "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Video_Item.py"
click node_video_info_item "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Video_Info_Item.py"
click node_download_item "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Download_Item.py"
click node_redis "https://github.com/kaifagou/scrapyspider/blob/master/RedisHelper/RedisHelper.py"
click node_save_pipeline "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/pipelines/SaveVideoPipeline.py"
click node_video_pipelines "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/pipelines/VideoRedisPipeline.py"
click node_info_pipelines "https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/pipelines/VideoInfoRedisPipeline.py"
click node_manager "https://github.com/kaifagou/scrapyspider/blob/master/Engine/Manager.py"
click node_spider_server "https://github.com/kaifagou/scrapyspider/blob/master/Engine/SpiderServer.py"
click node_mysql_helpers "https://github.com/kaifagou/scrapyspider/blob/master/Engine/MySqlHelper.py"
click node_homepage_mysql "https://github.com/kaifagou/scrapyspider/blob/master/Engine/HomepageToMysql.py"
click node_video_mysql "https://github.com/kaifagou/scrapyspider/blob/master/Engine/VideoToMysql.py"
click node_video_update "https://github.com/kaifagou/scrapyspider/blob/master/Engine/UpdateVideoToMysql.py"
click node_content_mysql "https://github.com/kaifagou/scrapyspider/blob/master/Engine/VideoContentFromMysql.py"
click node_info_mysql "https://github.com/kaifagou/scrapyspider/blob/master/Engine/VideoInfoFromMysql.py"
click node_ads_mysql "https://github.com/kaifagou/scrapyspider/blob/master/Engine/AdsFromMysql.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scheduler,node_adspy,node_adsector,node_homepage_spider,node_get_video_handler,node_video_spiders,node_video_info_spider,node_download_spider,node_ad_spiders toneBlue
class node_homepage_data,node_video_item,node_video_info_item,node_download_item,node_redis,node_save_pipeline,node_video_pipelines,node_info_pipelines,node_mysql toneAmber
class node_manager,node_spider_server,node_mysql_helpers,node_homepage_mysql,node_video_mysql,node_video_update,node_content_mysql,node_info_mysql,node_ads_mysql toneMint
class node_operator,node_external_fb toneIndigo
```

## Components

### Scraping workflows

- [Homepage scheduler](https://github.com/kaifagou/scrapyspider/blob/master/homepage.py): `homepage.py`
- [Adspy spider](https://github.com/kaifagou/scrapyspider/blob/master/startSpider/AdspySpider.py): `startSpider/AdspySpider.py`
- [AdSector spider](https://github.com/kaifagou/scrapyspider/blob/master/startSpider/AdSectorSpider.py): `startSpider/AdSectorSpider.py`
- [Homepage spider](https://github.com/kaifagou/scrapyspider/blob/master/startSpider/HomepageSpider.py): `startSpider/HomepageSpider.py`
- [Video detail handler](https://github.com/kaifagou/scrapyspider/blob/master/startSpider/GetVideoSpider.py): `startSpider/GetVideoSpider.py`
- [Video spiders](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/VideoSpider.py): `scrapyuniversal/spiders/VideoSpider.py`
- [Video info spider](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/VideoInfoSpider.py): `scrapyuniversal/spiders/VideoInfoSpider.py`
- [Media download spider](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/SaveVideoSpider.py): `scrapyuniversal/spiders/SaveVideoSpider.py`
- [Ad collection spiders](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/spiders/AdSectorSpider.py): `scrapyuniversal/spiders/AdSectorSpider.py`

### Items and queues

- [Homepage item](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Homepage_Item.py): `scrapyuniversal/items/Fb_Homepage_Item.py`
- [Video item](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Video_Item.py): `scrapyuniversal/items/Fb_Video_Item.py`
- [Video info item](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Video_Info_Item.py): `scrapyuniversal/items/Fb_Video_Info_Item.py`
- [Download item](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/items/Fb_Download_Item.py): `scrapyuniversal/items/Fb_Download_Item.py`
- [Redis queues](https://github.com/kaifagou/scrapyspider/blob/master/RedisHelper/RedisHelper.py): `RedisHelper/RedisHelper.py`
- [Media save pipeline](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/pipelines/SaveVideoPipeline.py): `scrapyuniversal/pipelines/SaveVideoPipeline.py`
- [Video pipelines](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/pipelines/VideoRedisPipeline.py): `scrapyuniversal/pipelines/VideoRedisPipeline.py`
- [Info pipelines](https://github.com/kaifagou/scrapyspider/blob/master/scrapyuniversal/pipelines/VideoInfoRedisPipeline.py): `scrapyuniversal/pipelines/VideoInfoRedisPipeline.py`

### Sync and storage

- [Refresh manager](https://github.com/kaifagou/scrapyspider/blob/master/Engine/Manager.py): `Engine/Manager.py`
- [Observer dispatcher](https://github.com/kaifagou/scrapyspider/blob/master/Engine/SpiderServer.py): `Engine/SpiderServer.py`
- [MySQL connection](https://github.com/kaifagou/scrapyspider/blob/master/Engine/MySqlHelper.py): `Engine/MySqlHelper.py`
- [Homepage persistence](https://github.com/kaifagou/scrapyspider/blob/master/Engine/HomepageToMysql.py): `Engine/HomepageToMysql.py`
- [Video persistence](https://github.com/kaifagou/scrapyspider/blob/master/Engine/VideoToMysql.py): `Engine/VideoToMysql.py`
- [Video updates](https://github.com/kaifagou/scrapyspider/blob/master/Engine/UpdateVideoToMysql.py): `Engine/UpdateVideoToMysql.py`
- [Media content sync](https://github.com/kaifagou/scrapyspider/blob/master/Engine/VideoContentFromMysql.py): `Engine/VideoContentFromMysql.py`
- [Video info sync](https://github.com/kaifagou/scrapyspider/blob/master/Engine/VideoInfoFromMysql.py): `Engine/VideoInfoFromMysql.py`
- [Ads sync](https://github.com/kaifagou/scrapyspider/blob/master/Engine/AdsFromMysql.py): `Engine/AdsFromMysql.py`

### Other components

- **Operator**
- **Facebook**
- **MySQL database**

## Connections

- Operator → Homepage scheduler: starts
- Homepage scheduler → Adspy spider: chains to (evidence: `homepage.py`)
- Homepage scheduler → AdSector spider: chains to (evidence: `homepage.py`)
- Homepage scheduler → Homepage spider: chains to (evidence: `homepage.py`)
- Homepage scheduler → Video detail handler: chains to (evidence: `homepage.py`)
- Homepage scheduler → Redis queues: connects (evidence: `homepage.py`)
- Adspy spider → Facebook: scrapes
- AdSector spider → Facebook: scrapes
- Homepage spider → Facebook: scrapes
- Video spiders → Facebook: scrapes
- Video info spider → Facebook: scrapes
- Media download spider → Facebook: downloads from
- Video detail handler → Video item: creates
- Video detail handler → Video info item: uses
- Video detail handler → Redis queues: uses
- Media save pipeline → Redis queues: queues results (evidence: `scrapyuniversal/pipelines/SaveVideoPipeline.py`)
- Video spiders → Video item: uses
- Video spiders → Video info item: uses
- Video pipelines → Redis queues: persists items
- Info pipelines → Redis queues: persists items
- Refresh manager → Redis queues: polls commands (evidence: `Engine/Manager.py`)
- Refresh manager → Observer dispatcher: dispatches (evidence: `Engine/Manager.py`)
- Refresh manager → Homepage persistence: registers observer (evidence: `Engine/Manager.py`)
- Refresh manager → Video updates: registers observer (evidence: `Engine/Manager.py`)
- Refresh manager → Media content sync: registers observer (evidence: `Engine/Manager.py`)
- Refresh manager → Video info sync: registers observer (evidence: `Engine/Manager.py`)
- Refresh manager → Ads sync: registers observer (evidence: `Engine/Manager.py`)
- Refresh manager → Video persistence: constructs (evidence: `Engine/Manager.py`)
- Refresh manager → Video info sync: constructs (evidence: `Engine/Manager.py`)
- Homepage persistence → MySQL connection: uses (evidence: `Engine/HomepageToMysql.py`)
- Homepage persistence → MySQL database: writes
- Video persistence → MySQL database: writes
- Video updates → MySQL database: updates
- Media content sync → MySQL database: reads
- Video info sync → MySQL database: reads
- Ads sync → MySQL database: reads
