How internetarchive/heritrix3 works
Project Type and Purpose
• Heritrix is an open-source, extensible, web-scale archival-quality web crawler.
• It is composed of:
• commons: shared utilities (BDB bindings, I/O, checkpointing, URI utilities)
• engine: core crawler framework (frontier, prefetch, processing pipeline, checkpointing, REST interface)
• modules: pluggable fetchers, extractors, canonicalizers, deciderules, writers, recrawl/revisit, seeds, net/robots policies
• contrib: optional extensions (AMQP/Kafka events, custom extractors, reporting)
• dist/docker/docs: packaging, deployment, documentation
File Structure
• Top-level:
• pom.xml: parent Maven project
• commons, engine, modules, contrib: four main Java sub-projects
• docker/: container setup (Dockerfile, compose, entrypoint)
• dist/: assembled binary distribution and scripts (heritrix launchers, configs, pagerank extras)
• docs/: Sphinx & custom extensions for user/operator guides
• engine/src/main/java/org/archive/crawler:
• framework: CrawlController, CrawlJob, Engine, ToePool/ToeThread (thread-pool)
• frontier: URI queues, BDB storage, cost & precedence policies
• prefetch: politeness, scoping, quota enforcement
• processor & postprocessor: link extraction, supplement scoping, reporting
• io: logging, error formatting
• restlet: REST API endpoints, web UI resources, Freemarker templates, static assets
• spring: bean factory, configuration overlays, lifecycle hooks
• modules/src/main/java/org/archive/modules:
• fetcher: HTTP/FTP/SFTP implementations, cookie stores, DNS, WhoIs, SSL
• extractor: HTML, PDF, SWF, XML, CSS, JavaScript, robots.txt, sitemap
• canonicalize & deciderules: URL normalization and filtering rules
• writer: ARC/WARC record builders, ARC/WARC writers, mirror writer
• recrawl/revisit, seeds, net (robots policies), writer pool
• contrib: AMQP & Kafka event feeds, specialized extractors
• commons: BDBJE integration, queue utilities, checkpointing support
Key Insights from README
• Heritrix runs as a long-lived Java service with:
• a REST API and web UI (built atop Restlet + Freemarker)
• a pluggable module pipeline (Maven packaging, Spring bean definitions)
• persistent frontier using BDBJE and optional external messaging (AMQP/Kafka)
• Configurable via XML/Groovy bean profiles and Spring classes
• Checkpointing & recoverability built into the core framework