How ngshiheng/burplist works
Main Components
a. Web Crawling Layer:
• Multiple Spiders (10+ craft beer sites)
• Middleware Components
• Item Pipelines
• Spider Locators
b. Data Processing Layer:
• Database Models
• Parsers
• Utilities
c. Infrastructure Layer:
• Database (PostgreSQL, indicated by alembic usage)
• External Services Integration
• Proxy Management System
[External Beer Websites]
• ↓
[Proxy Layer (ScraperAPI)]
• ↓
[Crawler Layer]
• Spider Components
• Middleware
• Item Pipeline
• ↓
[Data Processing Layer]
• Models
• Parsers
• ↓
[Database Layer]
• ↓
[Monitoring Services]
• Sentry
• ScrapeOps
Component Details
a. Spider Layer (Top):
• Them to their respective locators
• Middleware interaction (delayed_requests, proxy, default)
b. Processing Layer (Middle):
• Item pipeline flow
• Data deduplication process
• Parser utilities
c. Storage Layer (Bottom):
• Database schema representation
• Alembic migration connection
Integration Points
Clearly mark:
• ScrapeOps integration
• Sentry error tracking
• Scraper API proxy service
• Database connections
Deployment Flow
A separate section showing:
• Docker container structure
• CI/CD pipeline flow (GitHub Actions)
• Database migration process
Key Relationships to Highlight:
• Spider → Locator → Item Pipeline flow
• Middleware interactions with spiders
• Database model relationships
• External service integrations
• Component types
• Data flow directions
• Integration points
• Environment boundaries
This system design should effectively communicate both the high-level architecture and the important implementation details of the Burplist crawler system.