How ahartel/basic-common-crawl-pipeline works
Key Components to Include:
Data Sources
• Common Crawl Dataset (CC-MAIN-2024-30)
• cluster.idx file
• Index files (cdx files)
• WARC files
Core Components
• Batcher Service
• Worker Service(s)
• RabbitMQ Message Queue
• Prometheus Metrics Server
• Data Sources -> Batcher -> Message Queue -> Workers -> Output
Component Details
a. Data Sources Section (Left side):
• cluster.idx file
• Index files (CDX)
• WARC files
b. Batcher Service (Center-left):
• Index file downloader
• URL filter (English + 200 status)
• Batch creator (200 URLs per batch)
• RabbitMQ publisher
c. Message Queue (Center):
• Input queue
• Management interface (port 15672)
• AMQP interface (port 5672)
d. Workers Section (Center-right):
• Multiple worker instances showing:
• WARC downloader
• Trafilatura text extractor
• Text filtering
• Tokenization
• Storage output
e. Monitoring Section (Bottom):
• Prometheus metrics server
• Metrics collection paths from both batcher and workers
Additional Elements
Annotations for:
• Language implementations (Rust/Python)
• Configuration parameters
• Port numbers
• Batch sizes
• Scaling indicators
Technical Details to Highlight
Callouts showing:
• URL filtering criteria
• Batch size limits (200 entries)
• Supported crawl versions
• Error handling points
• Monitoring metrics