How apache/stormcrawler works
Project type and purpose
• This is Apache StormCrawler, an open-source Java framework for building low-latency, scalable web crawlers on top of Apache Storm.
• It is organized as a Maven multi-module project:
• core (the main library containing spouts, bolts, filters, protocols, schedulers, etc.)
• archetype (a project template for end-users)
• external modules (pluggable adapters for AWS, Solr, OpenSearch, SQL, Tika, WARC, Playwright, URL frontier, etc.)
High-level architecture
At runtime a StormCrawler topology executes on an Apache Storm cluster. The key runtime components are:
• Spouts (URL providers)
– AbstractQueryingSpout (pulls URLs from a status store)
– FileSpout / MemorySpout (seed lists)
• Fetcher bolts
– SimpleFetcherBolt / FetcherBolt (download pages via HTTP, file, Selenium, Playwright)
• Parser bolts
– JSoupParserBolt, FeedParserBolt, SiteMapParserBolt (extract links, parse content)
• Filtering & partitioning
– URLFilterBolt (applies regex, host, robots, depth filters)
– URLPartitionerBolt (stream groupings based on URL hash or host)
• Indexing bolts
– AbstractIndexerBolt implementations in core (StdOutIndexer, DummyIndexer)
– External indexers (SolrIndexerBolt, OpenSearchIndexerBolt, CloudSearchIndexerBolt, SQLIndexerBolt)
• Status update & scheduling
– StatusUpdaterBolt (updates URL status in store: memory, SQL, OpenSearch, URL frontier gRPC)
– Scheduler / URLBuffer (DefaultScheduler, AdaptiveScheduler, SimpleURLBuffer, PriorityURLBuffer)
External services & data stores invoked by the topology:
– Apache Storm cluster (Nimbus, Supervisors, Zookeeper)
– Indexing back-ends: Solr, Elasticsearch, OpenSearch, AWS CloudSearch, relational DB
– Caching back-ends: AWS S3, HDFS for WARC output
– Tika for advanced content parsing
Key architectural patterns
• Stream-processing pipeline: tuples flow from spout → fetch → parse → filter → index → status update → back to spout
• Plugin / reflection-based loading: components implement Configurable, loaded via class names in crawler-conf.yaml
• Maven multi-module and archetype: separation of core vs. adapters vs. project template
• External modules (grouped by type: AWS, Solr, OpenSearch, SQL, Tika, WARC, Playwright, URL frontier)
• Storm cluster (Nimbus, Supervisors, ZK)
• Back-end systems: Solr/Elasticsearch/OpenSearch clusters, SQL DB, AWS S3, HDFS, CloudSearch, Tika config/service
– Seed → Spout → Fetcher → Parser → Filter → Partitioner → Indexer → StatusUpdater → Scheduler → Spout
– Indexer → External index store
– Protocol → HTTP/Selenium/Playwright → External web servers
– Caching bolt → S3/HDFS
c. Patterns & labels: