How zone-eu/zone-mta works
Project type and purpose
• ZoneMTA is an outbound SMTP relay (MTA/MSA) built in Node.js. It accepts mail via SMTP or HTTP API, stores messages in MongoDB GridFS, and delivers them to remote SMTP servers (or HTTP endpoints) with fine-grained routing and retry logic. It uses Redis for locks/counters and has a plugin system for extensibility.
File structure
• Top-level files:
• app.js – the master process that boots the system
• bin/check-bounce.js – CLI utility for parsing bounce messages
• config/default.js – base configuration (zones, pools, ports, etc.)
• lib/ – core modules (SMTP interface, API server, parser, queue server, delivery logic, plugin handler, DB, connection pool, counters, DKIM, etc.)
• lib/receiver/ – incoming SMTP proxy
• lib/transport/ – low-level framing, client/server transport abstractions
• plugins/ – built-in core plugins (DKIM, bounce handling, spam checks, custom rewrites, etc.)
• services/receiver.js & services/sender.js – scripts that launch worker roles (SMTP/HTTP feeder and delivery workers)
• test/ – unit tests for most subsystems
• Config directory and indexes.yaml (for App Engine) hint at deployment settings. Gruntfile.js and GitHub workflows show build/test/release pipelines.
README for architecture insights
• There are two main pipelines:
a. Incoming pipeline: SMTP/HTTP → stream parsing → plugin hooks → GridFS storage → queue entries in MongoDB
b. Outgoing pipeline: queue poller → delivery workers (child processes) → plugin hooks → transport layer → remote SMTP or HTTP
• It lists external dependencies: MongoDB, Redis, DNS resolver, external Rspamd/spam detectors, Prometheus for metrics.
a. Main components to represent:
1. SMTP Frontend (lib/receiver/smtp-proxy.js)
2. HTTP API server (lib/api-server.js)
3. Stream Processor & Parser (lib/message-parser.js, lib/line-ends.js)
4. Plugin Engine (lib/plugin-handler.js + plugins/*)
5. Mail Drop & Queue Writer (GridFS via lib/db.js + lib/mail-queue.js)
6. MongoDB (GridFS storage + zone-queue collection)
7. Redis (locks, counters via lib/queue-locker.js, lib/ttl-cache.js, lib/counters.js)
8. Queue Server / Master (lib/queue-server.js)
9. Worker Processes (services/sender.js)
10. Connection Pool (lib/connection-pool.js)
11. Transport Layer (lib/transport/*)
12. DNS caching / resolver
13. External Services: remote SMTP servers, HTTP endpoints, Rspamd
14. Metrics exporter (lib/counters.js → Prometheus endpoint)
b. Interactions & data flows:
• From Queue Server to Worker → Plugin Engine (sender hooks) → Connection Pool → Transport → Remote SMTP/HTTP
• MongoDB interactions: GridFS write/read; queue reads/writes
• DNS lookups (caching in Redis or external) used by transport
• Metrics scraped by Prometheus from HTTP API
c. Architectural patterns & principles:
• Event-driven, streaming pipelines (no full in-memory load for large messages)
• Plugin (hook) system for extensibility at defined pipeline stages (feeder, sender, bounce, etc.)
• Master-worker model: single master (app.js) with child processes for delivery isolation and at-least-once delivery semantics
• Queued, retryable at-least-once semantics with locks stored in Redis, durable storage in MongoDB
• Connection pooling for efficiency, zone-based routing for IP warm-up and throttling
d. Key technologies/frameworks:
• Node.js core modules & streams
• MongoDB GridFS (mail.files, mail.chunks) + zone-queue collection
• Redis for locks (queue-locker), TTL cache, counters (Prometheus)
• lib/transport (custom frame parser, client/server) for master-child IPC
• Prometheus metrics endpoint
• Built-in plugins: DKIM signing, Rspamd, bounce handling, header rewrites