How betterment/delayed works
Project Type and Purpose
• This is an open-source Ruby gem (“delayed”) that implements a SQL-driven, multi-threaded ActiveJob backend for Rails. It replaces/extends older delayed_job backends, providing guaranteed at-least-once execution, co-transactional enqueues, priority tiers, built-in instrumentation, and a monitoring process.
File Structure Analysis
• Top-level:
• lib/ – core implementation under lib/delayed/… plus adapters, worker, monitor, plugins, serialization, engine, rake tasks.
• app/models/delayed/job.rb – ActiveRecord model representing enqueued jobs.
• lib/generators/delayed/… – Rails generators for migration and template.
• spec/ – comprehensive tests organized to mirror lib/ and app/ structure.
• Configuration files: .rubocop.yml, .rspec, CI workflows, Gemfile, delayed.gemspec, README.md.
README Insights
• Explains gem goals: reliability, scalability, resilience, maintainability.
• Describes two main runtime processes:
a. Worker – picks jobs (SQL SKIP LOCKED), runs each in a thread pool via concurrent-ruby.
b. Monitor – aggregates queue metrics and emits ActiveSupport notifications.
• Highlights extensibility: plugins, lifecycle hooks, ActiveSupport::Notifications events, custom adapters.
a. Rails Application (user code)
– Uses ActiveJob API to enqueue jobs (perform_later)
– Rails initializer sets config.active_job.queue_adapter = :delayed
b. ActiveRecord Backend
– Delayed::Job model mapped to delayed_jobs SQL table
– Co-transactional enqueue via AR transaction
c. Worker Process (rake delayed:work)
– Claims N jobs per cycle (Worker.max_claims) using optimized SQL pickup
– Spawns thread pool (concurrent-ruby)
– Wraps each job in Delayed::JobWrapper, applies plugins (connection, instrumentation), lifecycle hooks
– Executes underlying ActiveJob perform or legacy .delay call
– Emits ActiveSupport::Notifications (delayed.job.run/error/failure/enqueue/reserve_jobs)
d. Monitor Process (rake delayed:monitor)
– Periodically queries job counts, ages, lock ages, errors, etc.
– Emits metrics events (delayed.job.count, future_count, etc.) grouped by queue & priority
e. Plugin Subsystem
– Pluggable pipeline stages (lib/delayed/plugins/*.rb)
– Connection management, instrumentation, task preparation
f. Serialization Layer
– YAML/JSON serialization extensions, ActiveRecord hook for job payloads
g. Generators & Rake Tasks
– Migration generator for delayed_jobs table
– Rake tasks for delayed:work, delayed:monitor, test, instrumentation
Relationships and Data Flows
1. User code → ActiveJob enqueue → Delayed::Job AR insert in SQL table
2. Worker polls DB → locks rows (SKIP LOCKED) → spawns threads
3. Thread executes job: deserialize payload → apply plugins & lifecycle hooks → call perform → update attempts/failed_at
4. Worker emits events → external monitoring (e.g. StatsD)
5. Monitor process separately queries DB → emits aggregate events → external monitoring
Architectural Patterns & Design Principles
• Adapter Pattern: ActiveJob adapter in lib/delayed/active_job_adapter.rb
• Plugin/Hook Pipeline: core lifecycle extensibility via lib/delayed/plugins
• Observer/Event Bus: ActiveSupport::Notifications for instrumentation
• Engine Isolation: Rails engine in lib/delayed/engine.rb mounts tasks & generators
• Concurrency Control: thread-pool via concurrent-ruby, database‐backed locking
5. Worker → JobWrapper → ActiveJob perform (user code)
6. Worker → External Metrics (StatsD) via ActiveSupport events
8. Monitor → External Metrics
9. Off to the side, group files/folders: lib/delayed, app/models, generators, rake tasks, specs – call out extensibility and installation steps.