How striver-ing/wechat-spider works
Project Type and Purpose
• This project is a specialized web crawler (“微信爬虫”) designed to capture and process WeChat articles and related data. It uses a “man-in-the-middle” method to intercept network traffic from client devices (Android, iPhone, Mac, Windows) configured to use a proxy.
• It automates tasks such as detecting new articles, fetching article details, capturing engagement metrics (read, like, comment numbers), and handles proxy configuration (via mitmproxy certificates).
Analyzing the File Structure
• Top-Level Files: Files such as Dockerfile, docker-compose.yml, and requirements.txt indicate that the project is containerized and packaged for ease of deployment.
• The “wechat-spider” directory contains the bulk of the application:
• Configuration: Files like config.py and config.yaml for setting up MySQL/Redis connections and other parameters.
• The “core” folder holds the main logic modules:
• capture_packet.py – likely responsible for intercepting network traffic.
• data_pipeline.py – processes and forwards data between stages.
• deal_data.py – handles business logic to process raw data.
• task_manager.py – manages task scheduling and tracking to ensure no task (e.g. crawling for a WeChat account) is missed.
• The “db” folder includes modules for database interactions (mysqldb.py and redisdb.py) to connect to MySQL (for persistent storage) and Redis (for task caching).
• The “utils” folder contains supporting code (logging, tools, and selector utilities) that assist the main modules.
• “create_tables.py” is used to setup the MySQL tables automatically based on configuration.
Key Components
a. External Components:
• User Devices & WeChat Clients: Phones and PCs that are routed through the proxy.
• Proxy / MitM: The intermediary proxy (mitmproxy) which intercepts traffic.
• External Databases: MySQL for persistent storage and Redis for caching.
b. Internal Application Layers:
• Configuration Layer: Loads parameters via config.py and config.yaml.
• Core Engine:
• Packet Capture Module (capture_packet.py) captures network packets from proxy.
• Data Pipeline (data_pipeline.py) that organizes the flow of captured data.
• Data Processing Module (deal_data.py) to clean and transform collected data.
• Task Manager (task_manager.py) which schedules and tracks tasks based on input from the “wechat_account_task” table.
• Database Access Layer:
• MySQL Connector (mysqldb.py) for interaction with persistent storage.
• Redis Connector (redisdb.py) for caching and reducing database load.
• Utility Modules:
• Logger, Tools, and Selector assist with debugging, configuration sanity checks, and ancillary operations.
c. Deployment/Container Layer:
• Dockerfile and docker-compose.yml indicate that the whole system may be deployed in containers. This container layer is also important to note for real-world operational architecture.
Relationships and Data Flow
• Start at the Application Entry (run.py): This script initializes configuration, sets up logging via utils, and starts core modules.
• Data Flow:
1. The proxy (configured on client devices) channels network traffic (from WeChat) to the crawler.
2. The capture_packet module intercepts and forwards data.
3. The data moves through the data_pipeline, then is processed by deal_data.
4. The task_manager ensures each WeChat account and corresponding article is handled via status flags.
5. Processed data is stored in MySQL (via mysqldb.py), with Redis (via redisdb.py) serving as a quick-access cache for tasks.
• The utilities (logging, tools) run throughout the process to help monitor, debug, and optimize the workflow.
• Containerization is used to package and deploy these services reliably.
Architectural Patterns and Design Principles
• Separation of Concerns: Different directories for configuration, core business logic, data access, and utilities.
• Modular Design: Each Python module (for packet capture, data processing, task management) encapsulates specific functionality.
• Service Decoupling: Use of Redis for task caching decouples heavy read/write interactions with MySQL.
• Containerization: Deployment via Docker promotes consistency across environments.