# apache/stormcrawler architecture

> Architecture diagram of the GitHub repository apache/stormcrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/stormcrawler
- Repository: https://github.com/apache/stormcrawler

## Overview

##### Project type and purpose
- This is Apache StormCrawler, an open-source Java framework for building low-latency, scalable web crawlers on top of Apache Storm.
- It is organized as a Maven multi-module project:
- core (the main library containing spouts, bolts, filters, protocols, schedulers, etc.)
- archetype (a project template for end-users)
- external modules (pluggable adapters for AWS, Solr, OpenSearch, SQL, Tika, WARC, Playwright, URL frontier, etc.)

##### High-level architecture
At runtime a StormCrawler topology executes on an Apache Storm cluster. The key runtime components are:
- Spouts (URL providers)
– AbstractQueryingSpout (pulls URLs from a status store)
– FileSpout / MemorySpout (seed lists)
- Fetcher bolts
– SimpleFetcherBolt / FetcherBolt (download pages via HTTP, file, Selenium, Playwright)
- Parser bolts
– JSoupParserBolt, FeedParserBolt, SiteMapParserBolt (extract links, parse content)
- Filtering & partitioning
– URLFilterBolt (applies regex, host, robots, depth filters)
– URLPartitionerBolt (stream groupings based on URL hash or host)
- Indexing bolts
– AbstractIndexerBolt implementations in core (StdOutIndexer, DummyIndexer)
– External indexers (SolrIndexerBolt, OpenSearchIndexerBolt, CloudSearchIndexerBolt, SQLIndexerBolt)
- Status update & scheduling
– StatusUpdaterBolt (updates URL status in store: memory, SQL, OpenSearch, URL frontier gRPC)
– Scheduler / URLBuffer (DefaultScheduler, AdaptiveScheduler, SimpleURLBuffer, PriorityURLBuffer)

External services & data stores invoked by the topology:
– Apache Storm cluster (Nimbus, Supervisors, Zookeeper)
– Indexing back-ends: Solr, Elasticsearch, OpenSearch, AWS CloudSearch, relational DB
– Caching back-ends: AWS S3, HDFS for WARC output
– Tika for advanced content parsing

##### Key architectural patterns
- Stream-processing pipeline: tuples flow from spout → fetch → parse → filter → index → status update → back to spout
- Plugin / reflection-based loading: components implement Configurable, loaded via class names in crawler-conf.yaml
- Maven multi-module and archetype: separation of core vs. adapters vs. project template

- External modules (grouped by type: AWS, Solr, OpenSearch, SQL, Tika, WARC, Playwright, URL frontier)
- Storm cluster (Nimbus, Supervisors, ZK)
- Back-end systems: Solr/Elasticsearch/OpenSearch clusters, SQL DB, AWS S3, HDFS, CloudSearch, Tika config/service
– Seed → Spout → Fetcher → Parser → Filter → Partitioner → Indexer → StatusUpdater → Scheduler → Spout
– Indexer → External index store
– Protocol → HTTP/Selenium/Playwright → External web servers
– Caching bolt → S3/HDFS
c. Patterns & labels:

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Maven Multi-Module
    subgraph "Maven Multi-Module Project"
        direction TB
        core_lib["Core (core)"]:::core
        archetype["Archetype (archetype)"]:::external
        external_mods["External Modules"]:::external
    end

    %% Core Pipeline
    subgraph "StormCrawler Core Pipeline" 
        direction TB
        subgraph "Spouts" 
            direction TB
            AQS["AbstractQueryingSpout"]:::core
            FS["FileSpout"]:::core
            MS["MemorySpout"]:::core
        end
        subgraph "Fetcher Bolts"
            direction TB
            SF["SimpleFetcherBolt"]:::core
            FB["FetcherBolt"]:::core
        end
        subgraph "Parser Bolts"
            direction TB
            JP["JSoupParserBolt"]:::core
            FP["FeedParserBolt"]:::core
            SP["SiteMapParserBolt"]:::core
        end
        subgraph "Filters"
            direction TB
            UF["URLFilterBolt"]:::core
        end
        subgraph "Partitioning"
            direction TB
            UP["URLPartitionerBolt"]:::core
        end
        subgraph "Indexing Bolts"
            direction TB
            AIB["AbstractIndexerBolt"]:::core
            SOI["StdOutIndexer"]:::core
            DI["DummyIndexer"]:::core
        end
        subgraph "Status Updaters"
            direction TB
            ASU["AbstractStatusUpdaterBolt"]:::core
            MSU["MemoryStatusUpdater"]:::core
            SOSU["StdOutStatusUpdater"]:::core
        end
        subgraph "Scheduler & URLBuffer"
            direction TB
            DS["DefaultScheduler"]:::core
            AS["AdaptiveScheduler"]:::core
            UB["URLBuffers"]:::core
        end
    end

    %% Protocols
    subgraph "Protocols" 
        direction TB
        ProtocolAPI["Protocol API"]:::core
        HTTP["HttpProtocol"]:::core
        FileProt["FileProtocol"]:::core
        Selenium["SeleniumProtocol"]:::core
        PlaywrightProt["PlaywrightProtocol"]:::external
    end

    %% External Adapter Modules
    subgraph "External Adapter Modules"
        direction TB
        AWS["AWS (CloudSearch, S3)"]:::external
        SolrExt["Solr Adapter"]:::external
        OSExt["OpenSearch Adapter"]:::external
        SQLExt["SQL Adapter"]:::external
        TikaExt["Tika ParserBolt"]:::external
        WARC["WARC Spout/Output"]:::external
        URLF["URL Frontier gRPC"]:::external
    end

    %% Infrastructure & Services
    subgraph "Infrastructure & External Services"
        direction TB
        StormCluster["Apache Storm Cluster"]:::infra
        ZK["Zookeeper"]:::infra
        SolrStore["Solr/Elasticsearch/OpenSearch"]:::infra
        SQLDB["SQL Database"]:::infra
        S3["AWS S3 / HDFS"]:::infra
        CloudSearch["AWS CloudSearch"]:::infra
        TikaSvc["Tika Service"]:::infra
        Web["External Web"]:::infra
    end

    %% Data Flow
    AQS -->|tuples| SF
    FS -->|tuples| SF
    MS -->|tuples| SF
    SF -->|download| ProtocolAPI
    FB -->|download| ProtocolAPI
    ProtocolAPI -->|HTTP/Selenium/Playwright| Web
    ProtocolAPI -->|file| FileProt
    HTTP --> ProtocolAPI
    FileProt --> ProtocolAPI
    Selenium --> ProtocolAPI
    PlaywrightProt --> ProtocolAPI

    SF --> JP
    SF --> FP
    SF --> SP
    JP --> UF
    FP --> UF
    SP --> UF
    UF --> UP
    UP --> AIB
    UP --> SOI
    UP --> DI
    AIB --> ASU
    SOI --> ASU
    DI --> ASU

    %% External indexers connections
    SolrExt -->|indexing API| SolrStore
    OSExt -->|indexing API| SolrStore
    AWS -->|CloudSearch API| CloudSearch
    SQLExt -->|SQL API| SQLDB

    %% Connect core indexers to external indexers
    AIB --> SolrExt
    AIB --> OSExt
    AIB --> AWS
    AIB --> SQLExt

    %% Status Updaters to stores
    ASU --> StormCluster
    MSU --> StormCluster
    SOSU --> StormCluster
    URLF --> StormCluster
    SolrExt --> StormCluster
    OSExt --> StormCluster
    SQLExt --> StormCluster

    ASU --> ZK
    URLF --> ZK

    ASU --> SQLDB
    ASU --> SolrStore
    ASU --> S3

    %% Scheduler feedback
    ASU --> DS
    DS --> AQS
    DS --> FS
    AS --> AQS
    AS --> MS
    UB --> AQS

    %% Archetype connection
    archetype --> core_lib

    %% Storm run
    core_lib --> StormCluster

    %% Click events
    click AQS "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AbstractQueryingSpout.java"
    click FS "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/spout/FileSpout.java"
    click MS "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/spout/MemorySpout.java"
    click FB "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java"
    click SF "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/SimpleFetcherBolt.java"
    click JP "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java"
    click FP "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/FeedParserBolt.java"
    click SP "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java"
    click UF "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/URLFilterBolt.java"
    click UP "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/URLPartitionerBolt.java"
    click AIB "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/indexing/AbstractIndexerBolt.java"
    click SOI "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/indexing/StdOutIndexer.java"
    click DI "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/indexing/DummyIndexer.java"
    click SolrExt "https://github.com/apache/stormcrawler/blob/main/external/solr/src/main/java/org/apache/stormcrawler/solr/bolt/IndexerBolt.java"
    click OSExt "https://github.com/apache/stormcrawler/blob/main/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/bolt/IndexerBolt.java"
    click AWS "https://github.com/apache/stormcrawler/blob/main/external/aws/src/main/java/org/apache/stormcrawler/aws/bolt/CloudSearchIndexerBolt.java"
    click SQLExt "https://github.com/apache/stormcrawler/blob/main/external/sql/src/main/java/org/apache/stormcrawler/sql/IndexerBolt.java"
    click ASU "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AbstractStatusUpdaterBolt.java"
    click MSU "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/MemoryStatusUpdater.java"
    click SOSU "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/StdOutStatusUpdater.java"
    click URLF "https://github.com/apache/stormcrawler/blob/main/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java"
    click SolrStore "https://github.com/apache/stormcrawler/blob/main/external/solr/src/main/java/org/apache/stormcrawler/solr/persistence/StatusUpdaterBolt.java"
    click OSExt "https://github.com/apache/stormcrawler/blob/main/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/persistence/StatusUpdaterBolt.java"
    click SQLExt "https://github.com/apache/stormcrawler/blob/main/external/sql/src/main/java/org/apache/stormcrawler/sql/StatusUpdaterBolt.java"
    click DS "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/DefaultScheduler.java"
    click AS "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AdaptiveScheduler.java"
    click UB "https://github.com/apache/stormcrawler/tree/main/core/src/main/java/org/apache/stormcrawler/persistence/urlbuffer"
    click ProtocolAPI "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/Protocol.java"
    click HTTP "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/httpclient/HttpProtocol.java"
    click FileProt "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/file/FileProtocol.java"
    click Selenium "https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java"
    click PlaywrightProt "https://github.com/apache/stormcrawler/blob/main/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java"
    click archetype "https://github.com/apache/stormcrawler/tree/main/archetype/src/main/resources/archetype-resources"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#000,stroke-width:1px
```
