# internetarchive/heritrix3 architecture

> Architecture diagram of the GitHub repository internetarchive/heritrix3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internetarchive/heritrix3
- Repository: https://github.com/internetarchive/heritrix3

## Overview

##### Project Type and Purpose
- Heritrix is an open-source, extensible, web-scale archival-quality web crawler.
- It is composed of:
- commons: shared utilities (BDB bindings, I/O, checkpointing, URI utilities)
- engine: core crawler framework (frontier, prefetch, processing pipeline, checkpointing, REST interface)
- modules: pluggable fetchers, extractors, canonicalizers, deciderules, writers, recrawl/revisit, seeds, net/robots policies
- contrib: optional extensions (AMQP/Kafka events, custom extractors, reporting)
- dist/docker/docs: packaging, deployment, documentation

##### File Structure
- Top-level:
- pom.xml: parent Maven project
- commons, engine, modules, contrib: four main Java sub-projects
- docker/: container setup (Dockerfile, compose, entrypoint)
- dist/: assembled binary distribution and scripts (heritrix launchers, configs, pagerank extras)
- docs/: Sphinx & custom extensions for user/operator guides
- engine/src/main/java/org/archive/crawler:
- framework: CrawlController, CrawlJob, Engine, ToePool/ToeThread (thread-pool)
- frontier: URI queues, BDB storage, cost & precedence policies
- prefetch: politeness, scoping, quota enforcement
- processor & postprocessor: link extraction, supplement scoping, reporting
- io: logging, error formatting
- restlet: REST API endpoints, web UI resources, Freemarker templates, static assets
- spring: bean factory, configuration overlays, lifecycle hooks
- modules/src/main/java/org/archive/modules:
- fetcher: HTTP/FTP/SFTP implementations, cookie stores, DNS, WhoIs, SSL
- extractor: HTML, PDF, SWF, XML, CSS, JavaScript, robots.txt, sitemap
- canonicalize & deciderules: URL normalization and filtering rules
- writer: ARC/WARC record builders, ARC/WARC writers, mirror writer
- recrawl/revisit, seeds, net (robots policies), writer pool
- contrib: AMQP & Kafka event feeds, specialized extractors
- commons: BDBJE integration, queue utilities, checkpointing support

##### Key Insights from README
- Heritrix runs as a long-lived Java service with:
- a REST API and web UI (built atop Restlet + Freemarker)
- a pluggable module pipeline (Maven packaging, Spring bean definitions)
- persistent frontier using BDBJE and optional external messaging (AMQP/Kafka)
- Configurable via XML/Groovy bean profiles and Spring classes
- Checkpointing & recoverability built into the core framework

## Diagram (Mermaid)

```mermaid
flowchart TB
    UI_REST["Web UI / REST API"]:::ui
    Engine_Core["Engine Core\n(CrawlController, Engine, ToePool, CheckpointService)"]:::core
    Frontier["Frontier\n(WorkQueue & BDBJE)"]:::pipeline
    Prefetch["Prefetch & Quota Enforcement"]:::pipeline
    Processing["Processing Pipeline\n(Canonicalize → Deciderules → Extractors → Postprocessors)"]:::pipeline
    Writers["Writers & Storage\n(ARC/WARC, Mirror)"]:::pipeline
    Commons["Commons & Shared Utilities"]:::util
    Modules["Modules\n(Fetchers, Extractors, Deciderules, Writers, Seeds, Net)"]:::modules
    Contrib["Contrib Extensions\n(AMQP/Kafka, Reporting)"]:::modules
    Config["Configuration\n(Spring Beans, XML/Groovy)"]:::config
    Packaging["Packaging & Deployment\n(Docker & Distribution Scripts)"]:::packaging
    Docs["Documentation"]:::packaging

    UI_REST -->|submits jobs, queries status| Engine_Core
    Engine_Core -->|enqueue seeds| Frontier
    Frontier -->|dispatch URIs| Prefetch
    Prefetch -->|fetch responses| Processing
    Processing -->|write records| Writers
    Processing -->|new URIs| Frontier
    Processing -->|emit events| Contrib
    Contrib -->|publish/subscribe| Frontier
    Contrib -->|publish/subscribe| Processing
    Engine_Core --> Commons
    Frontier --> Commons
    Prefetch --> Commons
    Modules --> Prefetch
    Modules --> Processing
    Engine_Core --> Modules
    Engine_Core --> Config
    Config --> Engine_Core
    Packaging -->|contains| Docs

    click UI_REST "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/restlet"
    click UI_REST "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/resources/org/archive/crawler/restlet"
    click Engine_Core "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/framework"
    click Frontier "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/frontier"
    click Prefetch "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/prefetch"
    click Processing "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/processor"
    click Processing "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/postprocessor"
    click Commons "https://github.com/internetarchive/heritrix3/tree/master/commons/src/main/java/org/archive"
    click Commons "https://github.com/internetarchive/heritrix3/tree/master/commons/src/main/java/org/archive/bdb"
    click Commons "https://github.com/internetarchive/heritrix3/tree/master/commons/src/main/java/org/archive/checkpointing"
    click Commons "https://github.com/internetarchive/heritrix3/tree/master/commons/src/main/java/org/archive/io"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/fetcher"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/extractor"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/canonicalize"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/deciderules"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/writer"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/recrawl"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/revisit"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/seeds"
    click Modules "https://github.com/internetarchive/heritrix3/tree/master/modules/src/main/java/org/archive/modules/net"
    click Contrib "https://github.com/internetarchive/heritrix3/tree/master/contrib/src/main/java/org/archive"
    click Contrib "https://github.com/internetarchive/heritrix3/tree/master/contrib/src/main/java/org/archive/crawler/event"
    click Contrib "https://github.com/internetarchive/heritrix3/tree/master/contrib/src/main/java/org/archive/crawler/frontier"
    click Contrib "https://github.com/internetarchive/heritrix3/tree/master/contrib/src/main/java/org/archive/crawler/prefetch"
    click Contrib "https://github.com/internetarchive/heritrix3/tree/master/contrib/src/main/java/org/archive/crawler/reporting"
    click Config "https://github.com/internetarchive/heritrix3/tree/master/engine/src/main/java/org/archive/crawler/spring"
    click Packaging "https://github.com/internetarchive/heritrix3/tree/master/docker/Dockerfile"
    click Packaging "https://github.com/internetarchive/heritrix3/blob/master/docker/docker-compose.yml"
    click Packaging "https://github.com/internetarchive/heritrix3/blob/master/docker/entrypoint.sh"
    click Packaging "https://github.com/internetarchive/heritrix3/tree/master/dist/src/main/bin"
    click Packaging "https://github.com/internetarchive/heritrix3/tree/master/dist/src/main/conf"
    click Docs "https://github.com/internetarchive/heritrix3/tree/master/docs/"
    click Docs "https://github.com/internetarchive/heritrix3/blob/master/docs/_ext/beandoc.py"

    classDef ui fill:#AED6F1,stroke:#217DBB;
    classDef core fill:#ABEBC6,stroke:#28B463;
    classDef pipeline fill:#FCF3CF,stroke:#F4D03F;
    classDef modules fill:#FAD7A0,stroke:#E67E22;
    classDef util fill:#D5D8DC,stroke:#808B96;
    classDef config fill:#F5B7B1,stroke:#C0392B;
    classDef packaging fill:#D6EAF8,stroke:#5DADE2;
```
