How internetarchive/zeno works
Zeno is a configurable web crawler and web-page archiver. Users start a single-URL or list crawl through the CLI; the controller coordinates seed intake, preprocessing, fetching/archiving, postprocessing, and crawl completion. The product also supports HQ and local-queue sources, headless capture, format- and site-specific extraction, seen checks, WARC recording, and operational monitoring. The diagram follows documented capabilities and visible imports/calls; unsampled subsystem wiring is omitted or marked unverified.
Main entry points: pipeline.go Pipeline controller · postprocessor.go Postprocessing · preprocessor.go URL preprocessing · worker.go Fetch workers
Components 25
Entry and control
- CLI commands
cmd/get_list.go - Program entry
main.go - Pipeline controller
internal/pkg/controler/pipeline.go
Seeds and filtering
- Seed reactor
internal/pkg/reactor/reactor.go - URL preprocessing
internal/pkg/preprocessor/preprocessor.go - Seen checking
internal/pkg/preprocessor/seencheck/seencheck.go - HQ source
internal/pkg/source/hq - Local queue
internal/pkg/source/lq
Fetch and archive
- Fetch workers
internal/pkg/archiver/worker.go - HTTP archiver
internal/pkg/archiver/general/archiver.go - Headless archiver
internal/pkg/archiver/headless/archiver.go - WARC recording
internal/pkg/archiver/warc.go
Extraction and output
- Postprocessing
internal/pkg/postprocessor/postprocessor.go - Asset extraction
internal/pkg/postprocessor/assets.go - Format extractors
internal/pkg/postprocessor/extractor - Object listings
internal/pkg/postprocessor/extractor/object_storage.go - Reddit API support
internal/pkg/postprocessor/sitespecific/reddit/api.go - Domain crawl rules
internal/pkg/postprocessor/domainscrawl/domainscrawl.go - Crawl finisher
internal/pkg/finisher/finisher.go
Operations
- Metrics
internal/pkg/stats/methods.go - Monitoring API
internal/pkg/api/api.go - Resource watchers
internal/pkg/controler/watchers - Consul registration
internal/pkg/consul/consul.go
Other components
- Crawler user
- Web resources
Connections 27
- Crawler user → Program entry: runs
- Program entry → CLI commands: invokes
main.go - CLI commands → Pipeline controller: starts crawl
cmd/get_list.go - Pipeline controller → Seed reactor: starts
- Pipeline controller → URL preprocessing: starts
internal/pkg/controler/pipeline.go - Pipeline controller → Postprocessing: starts
internal/pkg/controler/pipeline.go - Pipeline controller → HQ source: selects source
internal/pkg/controler/pipeline.go - Pipeline controller → Local queue: selects source
internal/pkg/controler/pipeline.go - Pipeline controller → Monitoring API: starts optionally
internal/pkg/controler/pipeline.go - Pipeline controller → Resource watchers: starts
internal/pkg/controler/pipeline.go - Pipeline controller → Consul registration: registers optionally
internal/pkg/controler/pipeline.go - Pipeline controller → Metrics: initializes
internal/pkg/controler/pipeline.go - URL preprocessing → Seen checking: calls when configured
internal/pkg/preprocessor/preprocessor.go - URL preprocessing → Fetch workers: dispatches items
- Fetch workers → HTTP archiver: dispatches HTTP
internal/pkg/archiver/worker.go - Fetch workers → Headless archiver: dispatches headless
internal/pkg/archiver/worker.go - HTTP archiver → Web resources: fetches
- Headless archiver → Web resources: captures
- HTTP archiver → WARC recording: records traffic
- Headless archiver → WARC recording: records traffic
- Postprocessing → Asset extraction: extracts assets
internal/pkg/postprocessor/assets.go - Asset extraction → Format extractors: uses
internal/pkg/postprocessor/assets.go - Format extractors → Object listings: delegates listings
internal/pkg/postprocessor/extractor/object_storage.go - Asset extraction → Reddit API support: handles Reddit
internal/pkg/postprocessor/assets.go - Postprocessing → Domain crawl rules: applies domain rules
- Postprocessing → Crawl finisher: forwards completed items
- Pipeline controller → Crawl finisher: coordinates completion