# zone-eu/zone-mta architecture

> Architecture diagram of the GitHub repository zone-eu/zone-mta, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zone-eu/zone-mta
- Repository: https://github.com/zone-eu/zone-mta

## Overview

##### Project type and purpose
- ZoneMTA is an outbound SMTP relay (MTA/MSA) built in Node.js. It accepts mail via SMTP or HTTP API, stores messages in MongoDB GridFS, and delivers them to remote SMTP servers (or HTTP endpoints) with fine-grained routing and retry logic. It uses Redis for locks/counters and has a plugin system for extensibility.

##### File structure
- Top-level files:
- app.js – the master process that boots the system
- bin/check-bounce.js – CLI utility for parsing bounce messages
- config/default.js – base configuration (zones, pools, ports, etc.)
- lib/ – core modules (SMTP interface, API server, parser, queue server, delivery logic, plugin handler, DB, connection pool, counters, DKIM, etc.)
- lib/receiver/ – incoming SMTP proxy
- lib/transport/ – low-level framing, client/server transport abstractions
- plugins/ – built-in core plugins (DKIM, bounce handling, spam checks, custom rewrites, etc.)
- services/receiver.js & services/sender.js – scripts that launch worker roles (SMTP/HTTP feeder and delivery workers)
- test/ – unit tests for most subsystems
- Config directory and indexes.yaml (for App Engine) hint at deployment settings. Gruntfile.js and GitHub workflows show build/test/release pipelines.

##### README for architecture insights
- There are two main pipelines:
a. Incoming pipeline: SMTP/HTTP → stream parsing → plugin hooks → GridFS storage → queue entries in MongoDB
b. Outgoing pipeline: queue poller → delivery workers (child processes) → plugin hooks → transport layer → remote SMTP or HTTP
- It lists external dependencies: MongoDB, Redis, DNS resolver, external Rspamd/spam detectors, Prometheus for metrics.

a. Main components to represent:
1. SMTP Frontend (lib/receiver/smtp-proxy.js)
2. HTTP API server (lib/api-server.js)
3. Stream Processor & Parser (lib/message-parser.js, lib/line-ends.js)
4. Plugin Engine (lib/plugin-handler.js + plugins/*)
5. Mail Drop & Queue Writer (GridFS via lib/db.js + lib/mail-queue.js)
6. MongoDB (GridFS storage + zone-queue collection)
7. Redis (locks, counters via lib/queue-locker.js, lib/ttl-cache.js, lib/counters.js)
8. Queue Server / Master (lib/queue-server.js)
9. Worker Processes (services/sender.js)
10. Connection Pool (lib/connection-pool.js)
11. Transport Layer (lib/transport/*)
12. DNS caching / resolver
13. External Services: remote SMTP servers, HTTP endpoints, Rspamd
14. Metrics exporter (lib/counters.js → Prometheus endpoint)
b. Interactions & data flows:
- From Queue Server to Worker → Plugin Engine (sender hooks) → Connection Pool → Transport → Remote SMTP/HTTP
- MongoDB interactions: GridFS write/read; queue reads/writes
- DNS lookups (caching in Redis or external) used by transport
- Metrics scraped by Prometheus from HTTP API
c. Architectural patterns & principles:
- Event-driven, streaming pipelines (no full in-memory load for large messages)
- Plugin (hook) system for extensibility at defined pipeline stages (feeder, sender, bounce, etc.)
- Master-worker model: single master (app.js) with child processes for delivery isolation and at-least-once delivery semantics
- Queued, retryable at-least-once semantics with locks stored in Redis, durable storage in MongoDB
- Connection pooling for efficiency, zone-based routing for IP warm-up and throttling
d. Key technologies/frameworks:
- Node.js core modules & streams
- MongoDB GridFS (mail.files, mail.chunks) + zone-queue collection
- Redis for locks (queue-locker), TTL cache, counters (Prometheus)
- lib/transport (custom frame parser, client/server) for master-child IPC
- Prometheus metrics endpoint
- Built-in plugins: DKIM signing, Rspamd, bounce handling, header rewrites

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layers and Processes
    subgraph "Input Layer"
        direction TB
        SMTPProxy["SMTP Proxy\n(lib/receiver/smtp-proxy.js)\nport 2525"]:::core
        HTTPAPI["HTTP API Server\n(lib/api-server.js)\nport 12080"]:::core
    end

    subgraph "Parsing & Plugin"
        direction TB
        Parser["Stream Parser\n(lib/message-parser.js\nlib/line-ends.js)"]:::core
        PluginEngine["Plugin Engine\n(lib/plugin-handler.js\nplugins/core/)"]:::core
    end

    subgraph "Storage Layer"
        direction TB
        MailDrop["Mail Drop (GridFS)\n(lib/mail-drop.js\nlib/db.js)"]:::storage
        QueueWriter["Queue Writer\n(lib/mail-queue.js)"]:::storage
        MongoDB["MongoDB\n(GridFS & zone-queue)"]:::storage
        Redis["Redis\n(locks/counters)\n(lib/queue-locker.js\nlib/ttl-cache.js\nlib/counters.js)"]:::storage
    end

    subgraph "Master & IPC"
        direction TB
        QueueServer["Queue Server\n(app.js\nlib/queue-server.js)\nport 12081"]:::core
        Transport["Transport Layer\n(lib/transport/client.js\nlib/transport/server.js\nlib/transport/frame-parser.js\nlib/transport/frames.js\nlib/transport/connection.js)"]:::core
    end

    subgraph "Worker Layer"
        direction TB
        DeliveryWorker["Delivery Worker\n(services/sender.js\nlib/sender.js)"]:::core
        ConnPool["Connection Pool\n(lib/connection-pool.js)"]:::core
    end

    subgraph "External Services"
        direction TB
        RemoteSMTP["Remote SMTP Servers"]:::external
        HTTPDest["Remote HTTP Endpoints"]:::external
        Rspamd["Rspamd\n(spam detector)"]:::external
        DNS["DNS Resolver"]:::external
        Prometheus["Prometheus\n(metrics scraper)"]:::external
    end

    %% Incoming Pipeline
    SMTPProxy -->|"SMTP msg"| Parser
    HTTPAPI -->|"HTTP mail"| Parser
    Parser -->|"parsed stream"| PluginEngine
    PluginEngine -->|"hooked message"| MailDrop
    MailDrop -->|"store msg"| MongoDB
    PluginEngine -->|"enqueue"| QueueWriter
    QueueWriter -->|"queue entry"| MongoDB

    %% Master & Worker IPC
    QueueServer -->|"poll queue"| MongoDB
    QueueServer -->|"assign job"| Transport
    Transport -->|"deliver job"| DeliveryWorker
    DeliveryWorker -->|"ack or error"| Transport
    Transport -->|"response"| QueueServer

    %% Outgoing Pipeline
    DeliveryWorker -->|"pre-send hook"| PluginEngine
    PluginEngine -->|"send request"| ConnPool
    ConnPool -->|"get connection"| DNS
    ConnPool -->|"deliver msg"| RemoteSMTP
    ConnPool -->|"deliver msg"| HTTPDest
    DeliveryWorker -->|"spam check"| Rspamd

    %% Storage Interactions
    DeliveryWorker -->|"load msg"| MongoDB
    DeliveryWorker -->|"locks/counters"| Redis
    Parser -->|"locks/counters"| Redis
    QueueServer -->|"locks/counters"| Redis

    %% Metrics
    Redis -->|"metrics"| Prometheus
    QueueServer -->|"metrics"| Prometheus

    %% Click Events
    click SMTPProxy "https://github.com/zone-eu/zone-mta/blob/master/lib/receiver/smtp-proxy.js"
    click HTTPAPI "https://github.com/zone-eu/zone-mta/blob/master/lib/api-server.js"
    click Parser "https://github.com/zone-eu/zone-mta/blob/master/lib/message-parser.js"
    click Parser "https://github.com/zone-eu/zone-mta/blob/master/lib/line-ends.js"
    click PluginEngine "https://github.com/zone-eu/zone-mta/blob/master/lib/plugin-handler.js"
    click PluginEngine "https://github.com/zone-eu/zone-mta/tree/master/plugins/core/"
    click MailDrop "https://github.com/zone-eu/zone-mta/blob/master/lib/mail-drop.js"
    click MailDrop "https://github.com/zone-eu/zone-mta/blob/master/lib/db.js"
    click QueueWriter "https://github.com/zone-eu/zone-mta/blob/master/lib/mail-queue.js"
    click QueueServer "https://github.com/zone-eu/zone-mta/blob/master/app.js"
    click QueueServer "https://github.com/zone-eu/zone-mta/blob/master/lib/queue-server.js"
    click DeliveryWorker "https://github.com/zone-eu/zone-mta/blob/master/services/sender.js"
    click DeliveryWorker "https://github.com/zone-eu/zone-mta/blob/master/lib/sender.js"
    click ConnPool "https://github.com/zone-eu/zone-mta/blob/master/lib/connection-pool.js"
    click Transport "https://github.com/zone-eu/zone-mta/blob/master/lib/transport/client.js"
    click Transport "https://github.com/zone-eu/zone-mta/blob/master/lib/transport/server.js"
    click Transport "https://github.com/zone-eu/zone-mta/blob/master/lib/transport/frame-parser.js"
    click Transport "https://github.com/zone-eu/zone-mta/blob/master/lib/transport/frames.js"
    click Transport "https://github.com/zone-eu/zone-mta/blob/master/lib/transport/connection.js"
    click Redis "https://github.com/zone-eu/zone-mta/blob/master/lib/queue-locker.js"
    click Redis "https://github.com/zone-eu/zone-mta/blob/master/lib/ttl-cache.js"
    click Redis "https://github.com/zone-eu/zone-mta/blob/master/lib/counters.js"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#024B8A,color:#024B8A;
    classDef storage fill:#E8F5D7,stroke:#5A7D00,color:#5A7D00;
    classDef external fill:#F0F0F0,stroke:#666666,color:#666666;
```
