How lixi5338619/weixin-spider works
Main System Components
a) Client Layer:
• Windows WeChat Client
• Web Interface (Flask-based, port 5000)
• File Helper Window (for automation)
b) Proxy Layer:
• Mitmproxy (port 8080)
• Proxy Configuration Component
c) Core Processing Layer:
• Monitor System
• Crawler API
• Task Scheduler
d) Data Storage Layer:
• MySQL Database (weixin_spider)
• Redis Cache
Component Relationships and Data Flow
[WeChat Client] → [Mitmproxy] → [Crawler API]
• ↓
[Web Interface] → [Task Scheduler] → [Monitor System]
• ↓
• [Data Storage Layer]
Specific Component Details
For each major component, include:
Web Interface (webapp/):
• Connection to templates
• Static assets
• View controllers
Proxy Layer:
• Parameter extraction (x-wechat-key, __biz, appmsg_token, pass_ticket)
• Connection to automation system
Core Processing:
• Article data processing
• Account information processing
• Task management
Key Integration Points
Highlight:
• Mitmproxy integration with WeChat client
• Automation system interaction with File Helper
• API endpoints for data collection
• Database connections
Technical Details
Annotations for:
• Port numbers (5000, 8080)
• Database schema reference
• Redis key patterns
• API endpoints
Architectural Patterns
Emphasize:
• MVC pattern in webapp
• Event-driven architecture for monitoring
• Proxy pattern for data interception
Top-level organization
[Client Layer]
• ↓
[Proxy Layer]
• ↓
[Processing Layer]
• ↓
[Storage Layer]
Notes for
• Daily collection limits (5000-8000 articles)
• Critical configuration requirements
• Error handling paths
This system design should clearly show the flow from WeChat client through the proxy, to data processing and storage, while highlighting the automation and monitoring aspects that make this system unique.
Remember to maintain a balance between detail and clarity, focusing on the core architectural components while providing enough detail to understand the system's operation.