How archiveteam/grab-site works
Main Components
a) Core Components:
• Dashboard Server (gs-server)
• Crawler Engine (based on wpull)
• URL Queue Manager
• Ignore Pattern Manager
• WARC File Generator
• Duplicate Detection System
b) Storage Components:
• SQLite Database (for URL queue)
• WARC Files Storage
• Control Files Storage
Component Relationships and Data Flow
• User → Dashboard (HTTP/WebSocket)
• Dashboard ↔ Crawler Instances (WebSocket)
• Crawler → URL Queue Manager (bidirectional)
• Crawler → Target Websites (HTTP requests)
• Crawler → WARC Generator (content streaming)
• Crawler → Ignore Pattern Manager (URL filtering)
• Crawler → Duplicate Detection (content analysis)
Architectural Pattern
Use a distributed architecture pattern showing:
• Multiple crawler instances can run simultaneously
• Central dashboard for monitoring
• Independent storage systems
Component Details
Dashboard Server:
• gs-server |
• WebSocket Server |
• HTTP Server |
• Crawl Monitor |
• Status Updates |
Crawler Engine:
• Crawler Instance |
• wpull Engine |
• HTTP Client |
• Link Extractor |
• Content Processor |
Key Technical Elements to Highlight
• Python 3.7/3.8 runtime
• WebSocket communication
• SQLite database for queue
• WARC file format
• Regular expressions for URL filtering
• Disk-based queue system