# ahartel/basic-common-crawl-pipeline architecture

> Architecture diagram of the GitHub repository ahartel/basic-common-crawl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahartel/basic-common-crawl-pipeline
- Repository: https://github.com/ahartel/basic-common-crawl-pipeline

## Overview

Key Components to Include:

##### Data Sources
- Common Crawl Dataset (CC-MAIN-2024-30)
- cluster.idx file
- Index files (cdx files)
- WARC files

##### Core Components
- Batcher Service
- Worker Service(s)
- RabbitMQ Message Queue
- Prometheus Metrics Server

- Data Sources -> Batcher -> Message Queue -> Workers -> Output

##### Component Details

a. Data Sources Section (Left side):
- cluster.idx file
- Index files (CDX)
- WARC files

b. Batcher Service (Center-left):
- Index file downloader
- URL filter (English + 200 status)
- Batch creator (200 URLs per batch)
- RabbitMQ publisher

c. Message Queue (Center):
- Input queue
- Management interface (port 15672)
- AMQP interface (port 5672)

d. Workers Section (Center-right):
- Multiple worker instances showing:
- WARC downloader
- Trafilatura text extractor
- Text filtering
- Tokenization
- Storage output

e. Monitoring Section (Bottom):
- Prometheus metrics server
- Metrics collection paths from both batcher and workers

##### Additional Elements

Annotations for:
- Language implementations (Rust/Python)
- Configuration parameters
- Port numbers
- Batch sizes
- Scaling indicators

##### Technical Details to Highlight

Callouts showing:
- URL filtering criteria
- Batch size limits (200 entries)
- Supported crawl versions
- Error handling points
- Monitoring metrics

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef external fill:#D3D3D3,stroke:#333
    classDef core fill:#B3D9FF,stroke:#333
    classDef queue fill:#90EE90,stroke:#333
    classDef monitoring fill:#FFB366,stroke:#333
    classDef future fill:#F0F0F0,stroke:#333,stroke-dasharray: 5 5

    %% Data Sources
    subgraph DS["Data Sources"]
        CC["Common Crawl Dataset"]:::external
        CC --> IDX["cluster.idx"]:::external
        CC --> CDX["Index Files (CDX)"]:::external
        CC --> WARC["WARC Files"]:::external
    end

    %% Batcher Service
    subgraph BS["Batcher Service"]
        IFD["Index File Downloader"]:::core
        UF["URL Filter<br>(English + 200 status)"]:::core
        BC["Batch Creator<br>(200 URLs/batch)"]:::core
        PUB["RabbitMQ Publisher"]:::core
        
        IFD --> UF --> BC --> PUB
    end
    click BS "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/python/batcher.py"

    %% Message Queue
    subgraph MQ["RabbitMQ"]
        Q["Input Queue<br>AMQP:5672"]:::queue
        MGT["Management<br>Port:15672"]:::queue
    end
    click MQ "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/python/rabbitmq.py"

    %% Workers
    subgraph WK["Worker Service"]
        WD["WARC Downloader"]:::core
        TE["Trafilatura<br>Text Extractor"]:::core
        TF["Text Filtering"]:::future
        TK["Tokenization"]:::future
        SO["Storage Output"]:::future
        
        WD --> TE --> TF --> TK --> SO
    end
    click WK "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/python/worker.py"
    click TE "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/rust/src/trafilatura.rs"

    %% Monitoring
    subgraph MON["Monitoring"]
        PROM["Prometheus<br>Metrics Server"]:::monitoring
    end
    click PROM "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/prometheus/scrape_configs.yml"

    %% Connections
    DS --> BS
    BS --> MQ
    MQ --> WK
    BS -.-> PROM
    WK -.-> PROM

    %% Legend
    subgraph Legend
        L1["External System"]:::external
        L2["Core Component"]:::core
        L3["Message Queue"]:::queue
        L4["Monitoring"]:::monitoring
        L5["Future Extension"]:::future
        L6["Data Flow"]
        L7["Metrics Flow"]
    end

    %% Core Library
    click CC "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/rust/src/commoncrawl.rs"
    click BS "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/rust/src/bin/batcher.rs"
    click WK "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/rust/src/bin/worker.rs"
    click MQ "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/rust/src/rabbitmq.rs"
    click PROM "https://github.com/ahartel/basic-common-crawl-pipeline/blob/main/rust/src/tracing_and_metrics.rs"
```
