How internetarchive/brozzler works
Main Components
a) Core Components:
• Chrome/Chromium Browser Instance
• RethinkDB (Database)
• Warcprox (Web Archive Proxy)
• Brozzler Worker(s)
• Brozzler Dashboard
• PyWB (Wayback Machine)
b) Supporting Components:
• YT-DLP Integration
• X11 Environment (Xvnc4/Xvfb)
• Service Registry
Component Relationships and Data Flow
Frontend Layer:
• Brozzler Dashboard (Port 8000)
• PyWB Wayback Interface (Port 8880)
Processing Layer:
• Brozzler Workers (Multiple instances possible)
• Chrome/Chromium Browser instances
• YT-DLP Media Processor
Storage Layer:
• RethinkDB (Crawl state management)
• WARC Files Storage
Specific Implementation Details
• Rectangles: Core services (Dashboard, Workers, Browser)
• Hexagons: External dependencies (Chrome, YT-DLP)
Component Details
Dashboard:
• Brozzler Dashboard |
• Job Management |
• Status Viewing |
• Port: 8000 |
Worker Architecture:
• Brozzler Worker |
• Browser Control |
• URL Queue Management |
• Media Capture |
Database Schema Hints:
RethinkDB
• Crawl State
• Service Registry
• Captures Table
Deployment View
How components can be:
• Run together (brozzler-easy mode)
• Distributed across machines
• Scaled horizontally
• Distributed nature of the system
• Browser-based crawling approach
• Integration with web archiving tools
• Real-time playback capabilities