# internetarchive/zeno architecture

> Architecture diagram of the GitHub repository internetarchive/zeno, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internetarchive/zeno
- Repository: https://github.com/internetarchive/zeno

## Overview

Zeno is a configurable web crawler and web-page archiver. Users start a single-URL or list crawl through the CLI; the controller coordinates seed intake, preprocessing, fetching/archiving, postprocessing, and crawl completion. The product also supports HQ and local-queue sources, headless capture, format- and site-specific extraction, seen checks, WARC recording, and operational monitoring. The diagram follows documented capabilities and visible imports/calls; unsampled subsystem wiring is omitted or marked unverified.

Main entry points: [`internal/pkg/controler/pipeline.go`](https://github.com/internetarchive/zeno/blob/main/internal/pkg/controler/pipeline.go) (Pipeline controller), [`internal/pkg/postprocessor/postprocessor.go`](https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/postprocessor.go) (Postprocessing), [`internal/pkg/preprocessor/preprocessor.go`](https://github.com/internetarchive/zeno/blob/main/internal/pkg/preprocessor/preprocessor.go) (URL preprocessing), [`internal/pkg/archiver/worker.go`](https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/worker.go) (Fetch workers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and control"]
  node_cli["CLI commands<br/>[get_list.go]"]
  node_main["Program entry<br/>[main.go]"]
  node_controller["Pipeline controller<br/>[pipeline.go]"]
end

subgraph group_intake["Seeds and filtering"]
  node_reactor["Seed reactor<br/>[reactor.go]"]
  node_preprocessor["URL preprocessing<br/>[preprocessor.go]"]
  node_seencheck["Seen checking<br/>[seencheck.go]"]
  node_hq["HQ source"]
  node_lq[("Local queue")]
end

subgraph group_crawl["Fetch and archive"]
  node_worker["Fetch workers<br/>[worker.go]"]
  node_general["HTTP archiver<br/>[archiver.go]"]
  node_headless["Headless archiver<br/>[archiver.go]"]
  node_warc["WARC recording<br/>[warc.go]"]
end

subgraph group_process["Extraction and output"]
  node_postprocessor["Postprocessing<br/>[postprocessor.go]"]
  node_assets["Asset extraction<br/>[assets.go]"]
  node_extractors["Format extractors"]
  node_objectstore["Object listings<br/>[object_storage.go]"]
  node_reddit["Reddit API support<br/>[api.go]"]
  node_domains["Domain crawl rules<br/>[domainscrawl.go]"]
  node_finisher["Crawl finisher<br/>[finisher.go]"]
end

subgraph group_ops["Operations"]
  node_stats["Metrics<br/>[methods.go]"]
  node_api["Monitoring API<br/>[api.go]"]
  node_watchers["Resource watchers"]
  node_consul["Consul registration<br/>[consul.go]"]
end

node_user(("Crawler user"))
node_web(("Web resources"))

node_user -->|"runs"| node_main
node_main -->|"invokes"| node_cli
node_cli -->|"starts crawl"| node_controller
node_controller -.->|"starts"| node_reactor
node_controller -->|"starts"| node_preprocessor
node_controller -->|"starts"| node_postprocessor
node_controller -->|"selects source"| node_hq
node_controller -->|"selects source"| node_lq
node_controller -->|"starts optionally"| node_api
node_controller -->|"starts"| node_watchers
node_controller -->|"registers optionally"| node_consul
node_controller -->|"initializes"| node_stats
node_preprocessor -->|"calls when configured"| node_seencheck
node_preprocessor -.->|"dispatches items"| node_worker
node_worker -->|"dispatches HTTP"| node_general
node_worker -->|"dispatches headless"| node_headless
node_general -.->|"fetches"| node_web
node_headless -.->|"captures"| node_web
node_general -.->|"records traffic"| node_warc
node_headless -.->|"records traffic"| node_warc
node_postprocessor -->|"extracts assets"| node_assets
node_assets -->|"uses"| node_extractors
node_extractors -->|"delegates listings"| node_objectstore
node_assets -->|"handles Reddit"| node_reddit
node_postprocessor -.->|"applies domain rules"| node_domains
node_postprocessor -.->|"forwards completed items"| node_finisher
node_controller -.->|"coordinates completion"| node_finisher

click node_cli "https://github.com/internetarchive/zeno/blob/main/cmd/get_list.go"
click node_main "https://github.com/internetarchive/zeno/blob/main/main.go"
click node_controller "https://github.com/internetarchive/zeno/blob/main/internal/pkg/controler/pipeline.go"
click node_reactor "https://github.com/internetarchive/zeno/blob/main/internal/pkg/reactor/reactor.go"
click node_preprocessor "https://github.com/internetarchive/zeno/blob/main/internal/pkg/preprocessor/preprocessor.go"
click node_seencheck "https://github.com/internetarchive/zeno/blob/main/internal/pkg/preprocessor/seencheck/seencheck.go"
click node_hq "https://github.com/internetarchive/zeno/tree/main/internal/pkg/source/hq"
click node_lq "https://github.com/internetarchive/zeno/tree/main/internal/pkg/source/lq"
click node_worker "https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/worker.go"
click node_general "https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/general/archiver.go"
click node_headless "https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/headless/archiver.go"
click node_warc "https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/warc.go"
click node_postprocessor "https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/postprocessor.go"
click node_assets "https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/assets.go"
click node_extractors "https://github.com/internetarchive/zeno/tree/main/internal/pkg/postprocessor/extractor"
click node_objectstore "https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/extractor/object_storage.go"
click node_reddit "https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/sitespecific/reddit/api.go"
click node_domains "https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/domainscrawl/domainscrawl.go"
click node_finisher "https://github.com/internetarchive/zeno/blob/main/internal/pkg/finisher/finisher.go"
click node_stats "https://github.com/internetarchive/zeno/blob/main/internal/pkg/stats/methods.go"
click node_api "https://github.com/internetarchive/zeno/blob/main/internal/pkg/api/api.go"
click node_watchers "https://github.com/internetarchive/zeno/tree/main/internal/pkg/controler/watchers"
click node_consul "https://github.com/internetarchive/zeno/blob/main/internal/pkg/consul/consul.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_main,node_controller,node_user toneBlue
class node_reactor,node_preprocessor,node_seencheck,node_hq,node_lq toneAmber
class node_worker,node_general,node_headless,node_warc toneMint
class node_postprocessor,node_assets,node_extractors,node_objectstore,node_reddit,node_domains,node_finisher toneRose
class node_stats,node_api,node_watchers,node_consul,node_web toneIndigo
```

## Components

### Entry and control

- [CLI commands](https://github.com/internetarchive/zeno/blob/main/cmd/get_list.go): `cmd/get_list.go`
- [Program entry](https://github.com/internetarchive/zeno/blob/main/main.go): `main.go`
- [Pipeline controller](https://github.com/internetarchive/zeno/blob/main/internal/pkg/controler/pipeline.go): `internal/pkg/controler/pipeline.go`

### Seeds and filtering

- [Seed reactor](https://github.com/internetarchive/zeno/blob/main/internal/pkg/reactor/reactor.go): `internal/pkg/reactor/reactor.go`
- [URL preprocessing](https://github.com/internetarchive/zeno/blob/main/internal/pkg/preprocessor/preprocessor.go): `internal/pkg/preprocessor/preprocessor.go`
- [Seen checking](https://github.com/internetarchive/zeno/blob/main/internal/pkg/preprocessor/seencheck/seencheck.go): `internal/pkg/preprocessor/seencheck/seencheck.go`
- [HQ source](https://github.com/internetarchive/zeno/tree/main/internal/pkg/source/hq): `internal/pkg/source/hq`
- [Local queue](https://github.com/internetarchive/zeno/tree/main/internal/pkg/source/lq): `internal/pkg/source/lq`

### Fetch and archive

- [Fetch workers](https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/worker.go): `internal/pkg/archiver/worker.go`
- [HTTP archiver](https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/general/archiver.go): `internal/pkg/archiver/general/archiver.go`
- [Headless archiver](https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/headless/archiver.go): `internal/pkg/archiver/headless/archiver.go`
- [WARC recording](https://github.com/internetarchive/zeno/blob/main/internal/pkg/archiver/warc.go): `internal/pkg/archiver/warc.go`

### Extraction and output

- [Postprocessing](https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/postprocessor.go): `internal/pkg/postprocessor/postprocessor.go`
- [Asset extraction](https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/assets.go): `internal/pkg/postprocessor/assets.go`
- [Format extractors](https://github.com/internetarchive/zeno/tree/main/internal/pkg/postprocessor/extractor): `internal/pkg/postprocessor/extractor`
- [Object listings](https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/extractor/object_storage.go): `internal/pkg/postprocessor/extractor/object_storage.go`
- [Reddit API support](https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/sitespecific/reddit/api.go): `internal/pkg/postprocessor/sitespecific/reddit/api.go`
- [Domain crawl rules](https://github.com/internetarchive/zeno/blob/main/internal/pkg/postprocessor/domainscrawl/domainscrawl.go): `internal/pkg/postprocessor/domainscrawl/domainscrawl.go`
- [Crawl finisher](https://github.com/internetarchive/zeno/blob/main/internal/pkg/finisher/finisher.go): `internal/pkg/finisher/finisher.go`

### Operations

- [Metrics](https://github.com/internetarchive/zeno/blob/main/internal/pkg/stats/methods.go): `internal/pkg/stats/methods.go`
- [Monitoring API](https://github.com/internetarchive/zeno/blob/main/internal/pkg/api/api.go): `internal/pkg/api/api.go`
- [Resource watchers](https://github.com/internetarchive/zeno/tree/main/internal/pkg/controler/watchers): `internal/pkg/controler/watchers`
- [Consul registration](https://github.com/internetarchive/zeno/blob/main/internal/pkg/consul/consul.go): `internal/pkg/consul/consul.go`

### Other components

- **Crawler user**
- **Web resources**

## Connections

- Crawler user → Program entry: runs
- Program entry → CLI commands: invokes (evidence: `main.go`)
- CLI commands → Pipeline controller: starts crawl (evidence: `cmd/get_list.go`)
- Pipeline controller → Seed reactor: starts
- Pipeline controller → URL preprocessing: starts (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → Postprocessing: starts (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → HQ source: selects source (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → Local queue: selects source (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → Monitoring API: starts optionally (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → Resource watchers: starts (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → Consul registration: registers optionally (evidence: `internal/pkg/controler/pipeline.go`)
- Pipeline controller → Metrics: initializes (evidence: `internal/pkg/controler/pipeline.go`)
- URL preprocessing → Seen checking: calls when configured (evidence: `internal/pkg/preprocessor/preprocessor.go`)
- URL preprocessing → Fetch workers: dispatches items
- Fetch workers → HTTP archiver: dispatches HTTP (evidence: `internal/pkg/archiver/worker.go`)
- Fetch workers → Headless archiver: dispatches headless (evidence: `internal/pkg/archiver/worker.go`)
- HTTP archiver → Web resources: fetches
- Headless archiver → Web resources: captures
- HTTP archiver → WARC recording: records traffic
- Headless archiver → WARC recording: records traffic
- Postprocessing → Asset extraction: extracts assets (evidence: `internal/pkg/postprocessor/assets.go`)
- Asset extraction → Format extractors: uses (evidence: `internal/pkg/postprocessor/assets.go`)
- Format extractors → Object listings: delegates listings (evidence: `internal/pkg/postprocessor/extractor/object_storage.go`)
- Asset extraction → Reddit API support: handles Reddit (evidence: `internal/pkg/postprocessor/assets.go`)
- Postprocessing → Domain crawl rules: applies domain rules
- Postprocessing → Crawl finisher: forwards completed items
- Pipeline controller → Crawl finisher: coordinates completion
