How wellle/rmq works
Project type and purpose
• This is a Go-based library (“rmq”) that provides a Redis-backed message queue.
• It is not a standalone service or full-stack app, but an embeddable messaging toolkit for producers, consumers, and queue management, including heartbeating and cleaning.
File structure
• Top-level Go sources:
• connection.go, redis_client.go, redis_wrapper.go — handle connecting to single Redis or cluster
• queue.go, queue_cluster_test.go — queue abstraction and cluster support
• consumer.go, batch_consumer.go — consumer interfaces and batch consumers
• cleaner.go — background queue cleaner
• deliveries.go, delivery.go — internal Delivery types (Ack, Reject, Push)
• state.go, stats.go — track connection/queue/consumer state and collect stats
• header.go — optional payload header injection/extraction
• errors.go, rand.go — utility functions
• example/ directory contains minimal end-to-end demos: producer, consumer, cleaner, returner, purger, handler, batch_consumer.
• .github/workflows/*.yml — CI for tests, benchmarks, code coverage and release.
Extract architectural insights from README
• Core components: Connection, Queue, Producer, Consumer (and BatchConsumer), Cleaner, Stats Collector, optional Header-wrapping, TestConnection/TestQueue for unit tests.
• External dependency: one or more Redis instances (standalone or a cluster).
• Patterns:
• Publisher–subscriber via Redis lists (“ready”, “unacked”, “rejected”).
• Heartbeat goroutine per connection.
• Prefetch goroutine per consuming queue.
• Cleaner process to recover unacked deliveries.
• Retry semantics built into Ack/Reject/Push.
a. Library Client (your Go application)
– Producer code (Publish, PublishBytes, PayloadWithHeader)
– Consumer code (AddConsumer, AddBatchConsumer, consumer funcs)
– Management calls (StartConsuming, StopConsuming, ReturnRejected, PurgeRejected/PurgeReady)
b. rmq Library Subsystems
– Connection Manager (handles Redis connections + heartbeats)
– Queue Manager (manages queue metadata & push-queue relationships)
– Delivery Engine (moves payloads between ready → unacked → (ack/reject/push) lists)
– Cleaner (background process that moves expired unacked back to ready)
– Stats Collector (CollectStats API gathering connection/queue/consumer stats)
– Header Injector/Extractor (optional wrapper around payloads)
c. External Services
– Redis Instance(s) or Redis Cluster (ready, unacked, rejected lists per queue + heartbeat keys)
– Error Channel (Go channel for background error notifications)
d. Test Harness
– TestConnection/TestQueue/TestDelivery (in-memory mocks replacing Redis)
Relationships and data flow
• Producer → Queue.Publish() → ready list (Redis)
• Consumer StartConsuming:
• Prefetch goroutine: moves up to N items from ready → unacked list → Go delivery channel → Consumer.
• Consumer calls Ack/Reject/Push → Redis (unacked list → either deleted, moved to rejected or pushed to another queue).
• Heartbeat goroutine: updates a Redis key every second → Fresh TTL → signals “live connection”
• Cleaner (separate process or example/cleaner) periodically scans heartbeat keys → for expired connections → moves items from unacked → ready
• StatsCollector queries Redis keys and internal state → returns aggregate stats
• Optional Header Injector adds HTTP headers into payload; Extractor unwraps on Consume
Architectural patterns and design principles
• Separation of concerns: Connection vs. Queue vs. Delivery vs. Cleaner
• Concurrency via goroutines: heartbeat, prefetch, retries in Ack/Reject/Push
• Resilience: built-in retry on ack failures, background error channel, StopConsuming for graceful shutdown
• Extensibility: push-queue chaining for retry pipelines, batch consumers
Final checklist
• Components: ClientProducer, ClientConsumer, ConnectionManager, QueueManager, DeliveryEngine, HeaderModule, StatsCollector, Cleaner, Redis
• Interactions: Publish, Prefetch, Ack/Reject/Push, Heartbeat, Clean up, Stats collection
• Error flows: background error channel with three error types