# betterment/delayed architecture

> Architecture diagram of the GitHub repository betterment/delayed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/betterment/delayed
- Repository: https://github.com/betterment/delayed

## Overview

##### Project Type and Purpose
- This is an open-source Ruby gem (“`delayed`”) that implements a SQL-driven, multi-threaded ActiveJob backend for Rails. It replaces/extends older `delayed_job` backends, providing guaranteed at-least-once execution, co-transactional enqueues, priority tiers, built-in instrumentation, and a monitoring process.

##### File Structure Analysis
- Top-level:
- `lib/` – core implementation under `lib/delayed/…` plus adapters, worker, monitor, plugins, serialization, engine, rake tasks.
- `app/models/delayed/job.rb` – ActiveRecord model representing enqueued jobs.
- `lib/generators/delayed/…` – Rails generators for migration and template.
- `spec/` – comprehensive tests organized to mirror `lib/` and `app/` structure.
- Configuration files: `.rubocop.yml`, `.rspec`, CI workflows, `Gemfile`, `delayed.gemspec`, `README.md`.

##### README Insights
- Explains gem goals: reliability, scalability, resilience, maintainability.
- Describes two main runtime processes:
a. **Worker** – picks jobs (SQL SKIP LOCKED), runs each in a thread pool via `concurrent-ruby`.
b. **Monitor** – aggregates queue metrics and emits ActiveSupport notifications.
- Highlights extensibility: plugins, lifecycle hooks, ActiveSupport::Notifications events, custom adapters.

a. **Rails Application** (user code)
– Uses ActiveJob API to enqueue jobs (`perform_later`)
– Rails initializer sets `config.active_job.queue_adapter = :delayed`
b. **ActiveRecord Backend**
– `Delayed::Job` model mapped to `delayed_jobs` SQL table
– Co-transactional enqueue via AR transaction
c. **Worker Process** (`rake delayed:work`)
– Claims N jobs per cycle (`Worker.max_claims`) using optimized SQL pickup
– Spawns thread pool (`concurrent-ruby`)
– Wraps each job in `Delayed::JobWrapper`, applies plugins (`connection`, `instrumentation`), lifecycle hooks
– Executes underlying ActiveJob `perform` or legacy `.delay` call
– Emits ActiveSupport::Notifications (`delayed.job.run/error/failure/enqueue/reserve_jobs`)
d. **Monitor Process** (`rake delayed:monitor`)
– Periodically queries job counts, ages, lock ages, errors, etc.
– Emits metrics events (`delayed.job.count`, `future_count`, etc.) grouped by queue & priority
e. **Plugin Subsystem**
– Pluggable pipeline stages (`lib/delayed/plugins/*.rb`)
– Connection management, instrumentation, task preparation
f. **Serialization Layer**
– YAML/JSON serialization extensions, ActiveRecord hook for job payloads
g. **Generators & Rake Tasks**
– Migration generator for `delayed_jobs` table
– Rake tasks for `delayed:work`, `delayed:monitor`, test, instrumentation

##### Relationships and Data Flows
1. User code → ActiveJob enqueue → `Delayed::Job` AR insert in SQL table
2. Worker polls DB → locks rows (SKIP LOCKED) → spawns threads
3. Thread executes job: deserialize payload → apply plugins & lifecycle hooks → call `perform` → update `attempts`/`failed_at`
4. Worker emits events → external monitoring (e.g. StatsD)
5. Monitor process separately queries DB → emits aggregate events → external monitoring

##### Architectural Patterns & Design Principles
- Adapter Pattern: ActiveJob adapter in `lib/delayed/active_job_adapter.rb`
- Plugin/Hook Pipeline: core lifecycle extensibility via `lib/delayed/plugins`
- Observer/Event Bus: ActiveSupport::Notifications for instrumentation
- Engine Isolation: Rails engine in `lib/delayed/engine.rb` mounts tasks & generators
- Concurrency Control: thread-pool via `concurrent-ruby`, database‐backed locking

5. Worker → JobWrapper → ActiveJob perform (user code)
6. Worker → External Metrics (StatsD) via ActiveSupport events
8. Monitor → External Metrics
9. Off to the side, group files/folders: lib/delayed, app/models, generators, rake tasks, specs – call out extensibility and installation steps.

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Main Application Layers
  subgraph "Rails Application"
    direction TB
    RailsApp["Rails App\n(ActiveJob API)"]:::extensibility
    ActiveJobAdapter["ActiveJob Adapter"]:::extensibility
    RailsEngine["Rails Engine"]:::extensibility
  end

  %% Persistence Layer
  subgraph "Persistence Layer"
    direction TB
    DelayedJobModel["Delayed::Job Model"]:::persistence
    DB["SQL Database\ndelayed_jobs"]:::persistence
  end

  %% Runtime Processes
  subgraph "Runtime Processes"
    direction TB
    WorkerProcess["Worker Process\n(rake delayed:work)"]:::runtime
    MonitorProcess["Monitor Process\n(rake delayed:monitor)"]:::runtime
  end

  %% Worker Internals
  subgraph "Worker Internals"
    direction TB
    ThreadPool["Thread Pool\n(concurrent-ruby)"]:::runtime
    JobPreparer["Backend Job Preparer"]:::runtime
    JobWrapper["JobWrapper & Hooks"]:::runtime
    PluginPipeline["Plugin Subsystem"]:::extensibility
  end

  %% Plugin Subsystem Details
  subgraph "Plugins"
    direction TB
    PluginBase["Plugin Base"]:::extensibility
    ConnectionPlugin["Connection Plugin"]:::extensibility
    InstrumentationPlugin["Instrumentation Plugin"]:::extensibility
  end

  %% Serialization Layer
  subgraph "Serialization Layer"
    direction TB
    ActiveRecordSerialization["AR Serialization Ext"]:::extensibility
    PsychExt["Psych Extension"]:::extensibility
    SyckExt["Syck Extension"]:::extensibility
    YamlExt["YAML Extension"]:::extensibility
  end

  %% Tools & Extensibility
  subgraph "Generators & Tasks"
    direction TB
    Generators["Generators"]:::extensibility
    RakeTasks["Rake Tasks"]:::extensibility
  end

  %% Core Entry & Specs
  MainEntrypoint["Main Gem Entry"]:::extensibility
  GemSpec["Gem Specification"]:::extensibility
  Version["Version Constant"]:::extensibility
  Specs["Test Suite (spec/)"]:::external

  %% External Metrics
  MetricsSink["Metrics Sink\n(StatsD/Prometheus)"]:::external

  %% Relationships
  RailsApp -->|"perform_later"| DelayedJobModel
  DelayedJobModel -->|INSERT| DB
  RailsApp -.->|"uses adapter"| ActiveJobAdapter
  RailsApp -.->|"mounts"| RailsEngine

  DB -->|SELECT ... SKIP LOCKED| WorkerProcess
  WorkerProcess --> ThreadPool
  ThreadPool --> JobPreparer
  JobPreparer --> JobWrapper
  JobWrapper --> PluginPipeline
  PluginPipeline --> PluginBase
  PluginBase --> ConnectionPlugin
  PluginBase --> InstrumentationPlugin
  JobWrapper -->|"deserialize & execute"| RailsApp

  WorkerProcess -->|"emit events"| MetricsSink
  DB -->|"aggregate metrics query"| MonitorProcess
  MonitorProcess -->|"emit metrics"| MetricsSink

  RailsEngine --> Generators
  RailsEngine --> RakeTasks
  RakeTasks --> WorkerProcess
  RakeTasks --> MonitorProcess

  MainEntrypoint --> RailsEngine
  MainEntrypoint --> ActiveJobAdapter
  GemSpec --> MainEntrypoint
  Version --> MainEntrypoint

  Specs --> MainEntrypoint
  Specs --> RailsApp
  Specs --> WorkerProcess
  Specs --> MonitorProcess

  DelayedJobModel --> ActiveRecordSerialization
  ActiveRecordSerialization --> PsychExt
  ActiveRecordSerialization --> SyckExt
  ActiveRecordSerialization --> YamlExt

  %% Click Events
  click RailsApp "https://github.com/betterment/delayed/blob/main/app/models/delayed/job.rb"
  click DelayedJobModel "https://github.com/betterment/delayed/blob/main/app/models/delayed/job.rb"
  click ActiveJobAdapter "https://github.com/betterment/delayed/blob/main/lib/delayed/active_job_adapter.rb"
  click RailsEngine "https://github.com/betterment/delayed/blob/main/lib/delayed/engine.rb"
  click WorkerProcess "https://github.com/betterment/delayed/blob/main/lib/delayed/worker.rb"
  click JobWrapper "https://github.com/betterment/delayed/blob/main/lib/delayed/job_wrapper.rb"
  click JobPreparer "https://github.com/betterment/delayed/blob/main/lib/delayed/backend/job_preparer.rb"
  click PluginBase "https://github.com/betterment/delayed/blob/main/lib/delayed/plugin.rb"
  click ConnectionPlugin "https://github.com/betterment/delayed/blob/main/lib/delayed/plugins/connection.rb"
  click InstrumentationPlugin "https://github.com/betterment/delayed/blob/main/lib/delayed/plugins/instrumentation.rb"
  click MonitorProcess "https://github.com/betterment/delayed/blob/main/lib/delayed/monitor.rb"
  click ActiveRecordSerialization "https://github.com/betterment/delayed/blob/main/lib/delayed/serialization/active_record.rb"
  click PsychExt "https://github.com/betterment/delayed/blob/main/lib/delayed/psych_ext.rb"
  click SyckExt "https://github.com/betterment/delayed/blob/main/lib/delayed/syck_ext.rb"
  click YamlExt "https://github.com/betterment/delayed/blob/main/lib/delayed/yaml_ext.rb"
  click RakeTasks "https://github.com/betterment/delayed/blob/main/lib/delayed/tasks.rb"
  click Generators "https://github.com/betterment/delayed/blob/main/lib/generators/delayed/generator.rb"
  click Generators "https://github.com/betterment/delayed/blob/main/lib/generators/delayed/migration_generator.rb"
  click Generators "https://github.com/betterment/delayed/blob/main/lib/generators/delayed/next_migration_version.rb"
  click Generators "https://github.com/betterment/delayed/blob/main/lib/generators/delayed/templates/migration.rb"
  click MainEntrypoint "https://github.com/betterment/delayed/blob/main/lib/delayed.rb"
  click GemSpec "https://github.com/betterment/delayed/blob/main/delayed.gemspec"
  click Version "https://github.com/betterment/delayed/blob/main/lib/delayed/version.rb"

  %% Styles
  classDef persistence fill:#9f9,stroke:#333,stroke-width:1px
  classDef runtime fill:#99f,stroke:#333,stroke-width:1px
  classDef extensibility fill:#ff9,stroke:#333,stroke-width:1px
  classDef external fill:#ccc,stroke:#333,stroke-width:1px
```
