# striver-ing/wechat-spider architecture

> Architecture diagram of the GitHub repository striver-ing/wechat-spider, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/striver-ing/wechat-spider
- Repository: https://github.com/striver-ing/wechat-spider

## Overview

##### Project Type and Purpose
- This project is a specialized web crawler (“微信爬虫”) designed to capture and process WeChat articles and related data. It uses a “man-in-the-middle” method to intercept network traffic from client devices (Android, iPhone, Mac, Windows) configured to use a proxy.
- It automates tasks such as detecting new articles, fetching article details, capturing engagement metrics (read, like, comment numbers), and handles proxy configuration (via mitmproxy certificates).

##### Analyzing the File Structure
- Top-Level Files: Files such as Dockerfile, docker-compose.yml, and requirements.txt indicate that the project is containerized and packaged for ease of deployment.
- The “wechat-spider” directory contains the bulk of the application:
- Configuration: Files like config.py and config.yaml for setting up MySQL/Redis connections and other parameters.
- The “core” folder holds the main logic modules:
- capture_packet.py – likely responsible for intercepting network traffic.
- data_pipeline.py – processes and forwards data between stages.
- deal_data.py – handles business logic to process raw data.
- task_manager.py – manages task scheduling and tracking to ensure no task (e.g. crawling for a WeChat account) is missed.
- The “db” folder includes modules for database interactions (mysqldb.py and redisdb.py) to connect to MySQL (for persistent storage) and Redis (for task caching).
- The “utils” folder contains supporting code (logging, tools, and selector utilities) that assist the main modules.
- “create_tables.py” is used to setup the MySQL tables automatically based on configuration.

##### Key Components
a. External Components:
- User Devices & WeChat Clients: Phones and PCs that are routed through the proxy.
- Proxy / MitM: The intermediary proxy (mitmproxy) which intercepts traffic.
- External Databases: MySQL for persistent storage and Redis for caching.
b. Internal Application Layers:
- Configuration Layer: Loads parameters via config.py and config.yaml.
- Core Engine:
- Packet Capture Module (capture_packet.py) captures network packets from proxy.
- Data Pipeline (data_pipeline.py) that organizes the flow of captured data.
- Data Processing Module (deal_data.py) to clean and transform collected data.
- Task Manager (task_manager.py) which schedules and tracks tasks based on input from the “wechat_account_task” table.
- Database Access Layer:
- MySQL Connector (mysqldb.py) for interaction with persistent storage.
- Redis Connector (redisdb.py) for caching and reducing database load.
- Utility Modules:
- Logger, Tools, and Selector assist with debugging, configuration sanity checks, and ancillary operations.
c. Deployment/Container Layer:
- Dockerfile and docker-compose.yml indicate that the whole system may be deployed in containers. This container layer is also important to note for real-world operational architecture.

##### Relationships and Data Flow
- Start at the Application Entry (run.py): This script initializes configuration, sets up logging via utils, and starts core modules.
- Data Flow:
1. The proxy (configured on client devices) channels network traffic (from WeChat) to the crawler.
2. The capture_packet module intercepts and forwards data.
3. The data moves through the data_pipeline, then is processed by deal_data.
4. The task_manager ensures each WeChat account and corresponding article is handled via status flags.
5. Processed data is stored in MySQL (via mysqldb.py), with Redis (via redisdb.py) serving as a quick-access cache for tasks.
- The utilities (logging, tools) run throughout the process to help monitor, debug, and optimize the workflow.
- Containerization is used to package and deploy these services reliably.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Different directories for configuration, core business logic, data access, and utilities.
- Modular Design: Each Python module (for packet capture, data processing, task management) encapsulates specific functionality.
- Service Decoupling: Use of Redis for task caching decouples heavy read/write interactions with MySQL.
- Containerization: Deployment via Docker promotes consistency across environments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Components
    subgraph "External Components"
        UD("User Devices & WeChat Clients"):::external
        Proxy("MitM Proxy"):::external
    end

    %% Application Entry
    subgraph "Application Entry"
        Entry("Application Entry (run.py)"):::entry
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        Config("Configuration (config.py, config.yaml)"):::config
    end

    %% Core Engine
    subgraph "Core Engine"
        PC("Packet Capture Module"):::core
        DP("Data Pipeline Module"):::core
        DProc("Data Processing Module"):::core
        TM("Task Manager Module"):::core
    end

    %% Database Access Layer
    subgraph "Database Access"
        MySQL("MySQL Access"):::db
        Redis("Redis Access"):::db
        Tables("MySQL Table Setup Script"):::db
    end

    %% Utility Layer
    subgraph "Utility Layer"
        Utils("Utility Modules"):::utils
    end

    %% Deployment/Containerization
    subgraph "Deployment/Containerization"
        Docker("Dockerfile"):::deploy
        Compose("docker-compose.yml"):::deploy
    end

    %% Connections
    UD -->|"RoutesTraffic"| Proxy
    Proxy -->|"InterceptsData"| PC

    Entry -->|"Initializes"| Config
    Entry -->|"StartsCore"| PC

    Config -->|"Configures"| PC
    Config -->|"Configures"| DP
    Config -->|"Configures"| DProc
    Config -->|"Configures"| TM

    PC -->|"FeedsData"| DP
    DP -->|"PassesData"| DProc
    DProc -->|"ProcessesTasks"| TM

    TM -->|"StoresData"| MySQL
    TM -->|"CachesData"| Redis

    Utils -->|"Supports"| PC
    Utils -->|"Supports"| DP
    Utils -->|"Supports"| DProc
    Utils -->|"Supports"| TM

    Docker -->|"Deploys"| Entry
    Compose -->|"Deploys"| Entry

    %% Click Events
    click Config "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/config.py"
    click Config "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/config.yaml"
    click PC "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/core/capture_packet.py"
    click DP "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/core/data_pipeline.py"
    click DProc "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/core/deal_data.py"
    click TM "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/core/task_manager.py"
    click MySQL "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/db/mysqldb.py"
    click Redis "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/db/redisdb.py"
    click Utils "https://github.com/striver-ing/wechat-spider/tree/master/wechat-spider/utils/"
    click Tables "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/create_tables.py"
    click Entry "https://github.com/striver-ing/wechat-spider/blob/master/wechat-spider/run.py"
    click Docker "https://github.com/striver-ing/wechat-spider/tree/master/Dockerfile"
    click Compose "https://github.com/striver-ing/wechat-spider/blob/master/docker-compose.yml"

    %% Styles
    classDef external fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef entry fill:#ffffff,stroke:#0000ff,stroke-width:2px;
    classDef config fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef core fill:#66ccff,stroke:#003366,stroke-width:2px;
    classDef db fill:#ff9999,stroke:#990000,stroke-width:2px;
    classDef utils fill:#ccccff,stroke:#333399,stroke-width:2px;
    classDef deploy fill:#ffcc99,stroke:#cc6600,stroke-width:2px;
```
