How antirez/disque works
Project Type and Purpose
• Disque is a distributed, in-memory job queue/message broker written in C.
• It mimics Redis’s non-blocking event-loop server model but adds synchronous replication, multi-master clustering, and at-least-once/at-most-once delivery semantics.
• Purpose: provide a fault-tolerant, self-contained, simple, high-performance job queue across multiple nodes.
Key Architectural Elements from File Structure
• Top‐level directories:
• src/ – core server implementation (networking, event loop, job/queue logic, cluster bus)
• deps/ – bundled libraries (hiredis client, jemalloc allocator, linenoise CLI helper)
• tests/ – integration and unit tests for cluster and server behavior
• Build and config: Makefile, disque.conf example, Travis CI setup
• No separate frontend; clients use Redis protocol via hiredis or custom libraries.
Main Components
a. Clients (Producers/Consumers)
• Via Redis‐compatible TCP protocol (default port 7711)
• Send ADDJOB, GETJOB, ACKJOB, FASTACK, etc.
b. Disque Nodes
• Event-driven server (disque-server)
• Contains modules: networking (ae.*), command parser, job store, queue manager, cluster bus, AOF persistence
c. Cluster Bus (Inter-node Communication)
• Full mesh TCP links (each node ↔ every other node) on port = basePort + 10000
• Exchanges PING/PONG/FAIL for failure detection, REPLJOB/GOTJOB for replication, ENQUEUE/QUEUED/WILLQUEUE for re-queue logic, SETACK/GOTACK/DELJOB for ack propagation and GC, NEEDJOBS/YOURJOBS for federation
d. In-memory Data Structures
• Job objects with state (“wait-repl”, “active”, “queued”, “acked”)
• Queues per topic (sorted by creation time)
• Tables for delivered replicas and ack confirmations
e. Optional Persistence
• AOF file for write-ahead logging of job creation/deletion
• Controlled restart mode for atomic reload of full state
Relationships and Data Flow
• Client → Node: commands flow over TCP; node processes commands in event loop
• Node ↔ Node (Cluster Bus):
• Replication messages (REPLJOB → GOTJOB) to satisfy W acknowledgements
• Queue federation (NEEDJOBS → YOURJOBS) when local queue empty but client blocked
• Ack propagation (ACKJOB → SETACK → GOTACK → DELJOB) to garbage‐collect jobs
• Re‐queue timers (WILLQUEUE/QUEUED) for retry semantics
• Node ↔ Disk: AOF writes on ADDJOB/DELJOB, and reload on startup or controlled rewrite
Architectural Patterns and Design Principles
• Event-driven, single‐threaded reactor pattern (ae.c/h modules)
• Share-nothing, multi-master cluster: no leader election, symmetric roles
• Full mesh gossip for failure detection (light-weight PING/PONG/FAIL)
• Best-effort, eventually consistent federation for queue federation
• Asynchronous replication with tunable sync parameter (W replicas)
• Timer‐based state machines for re-queue and garbage collection