# wellle/rmq architecture

> Architecture diagram of the GitHub repository wellle/rmq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wellle/rmq
- Repository: https://github.com/wellle/rmq

## Overview

##### Project type and purpose
- This is a Go-based library (“rmq”) that provides a Redis-backed message queue.
- It is not a standalone service or full-stack app, but an embeddable messaging toolkit for producers, consumers, and queue management, including heartbeating and cleaning.

##### File structure
- Top-level Go sources:
- connection.go, redis_client.go, redis_wrapper.go — handle connecting to single Redis or cluster
- queue.go, queue_cluster_test.go — queue abstraction and cluster support
- consumer.go, batch_consumer.go — consumer interfaces and batch consumers
- cleaner.go — background queue cleaner
- deliveries.go, delivery.go — internal Delivery types (Ack, Reject, Push)
- state.go, stats.go — track connection/queue/consumer state and collect stats
- header.go — optional payload header injection/extraction
- errors.go, rand.go — utility functions
- example/ directory contains minimal end-to-end demos: producer, consumer, cleaner, returner, purger, handler, batch_consumer.
- .github/workflows/*.yml — CI for tests, benchmarks, code coverage and release.

##### Extract architectural insights from README
- Core components: Connection, Queue, Producer, Consumer (and BatchConsumer), Cleaner, Stats Collector, optional Header-wrapping, TestConnection/TestQueue for unit tests.
- External dependency: one or more Redis instances (standalone or a cluster).
- Patterns:
- Publisher–subscriber via Redis lists (“ready”, “unacked”, “rejected”).
- Heartbeat goroutine per connection.
- Prefetch goroutine per consuming queue.
- Cleaner process to recover unacked deliveries.
- Retry semantics built into Ack/Reject/Push.

a. Library Client (your Go application)
– Producer code (Publish, PublishBytes, PayloadWithHeader)
– Consumer code (AddConsumer, AddBatchConsumer, consumer funcs)
– Management calls (StartConsuming, StopConsuming, ReturnRejected, PurgeRejected/PurgeReady)
b. rmq Library Subsystems
– Connection Manager (handles Redis connections + heartbeats)
– Queue Manager (manages queue metadata & push-queue relationships)
– Delivery Engine (moves payloads between ready → unacked → (ack/reject/push) lists)
– Cleaner (background process that moves expired unacked back to ready)
– Stats Collector (CollectStats API gathering connection/queue/consumer stats)
– Header Injector/Extractor (optional wrapper around payloads)
c. External Services
– Redis Instance(s) or Redis Cluster (ready, unacked, rejected lists per queue + heartbeat keys)
– Error Channel (Go channel for background error notifications)
d. Test Harness
– TestConnection/TestQueue/TestDelivery (in-memory mocks replacing Redis)

##### Relationships and data flow
- Producer → Queue.Publish() → ready list (Redis)
- Consumer StartConsuming:
- Prefetch goroutine: moves up to N items from ready → unacked list → Go delivery channel → Consumer.
- Consumer calls Ack/Reject/Push → Redis (unacked list → either deleted, moved to rejected or pushed to another queue).
- Heartbeat goroutine: updates a Redis key every second → Fresh TTL → signals “live connection”
- Cleaner (separate process or example/cleaner) periodically scans heartbeat keys → for expired connections → moves items from unacked → ready
- StatsCollector queries Redis keys and internal state → returns aggregate stats
- Optional Header Injector adds HTTP headers into payload; Extractor unwraps on Consume

##### Architectural patterns and design principles
- Separation of concerns: Connection vs. Queue vs. Delivery vs. Cleaner
- Concurrency via goroutines: heartbeat, prefetch, retries in Ack/Reject/Push
- Resilience: built-in retry on ack failures, background error channel, StopConsuming for graceful shutdown
- Extensibility: push-queue chaining for retry pipelines, batch consumers

##### Final checklist
- Components: ClientProducer, ClientConsumer, ConnectionManager, QueueManager, DeliveryEngine, HeaderModule, StatsCollector, Cleaner, Redis
- Interactions: Publish, Prefetch, Ack/Reject/Push, Heartbeat, Clean up, Stats collection
- Error flows: background error channel with three error types

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client Application"
        CP["ClientProducer"]:::client
        CC["ClientConsumer"]:::client
        CBC["ClientBatchConsumer"]:::client
        CR["ExampleReturner"]:::client
        CPg["ExamplePurger"]:::client
        CH["ExampleHandler"]:::client
    end

    subgraph "Library Subsystems"
        CM["ConnectionManager"]:::subsystem
        QM["QueueManager"]:::subsystem
        DE["DeliveryEngine"]:::subsystem
        Cleaner["Cleaner"]:::subsystem
        SC["StatsCollector"]:::subsystem
        HM["HeaderModule"]:::subsystem
        UF["UtilityFunctions"]:::subsystem
        SH["StateHeartbeat"]:::subsystem
    end

    subgraph "Test Harness"
        TC["TestConnection"]:::test
        TQ["TestQueue"]:::test
        TD["TestDelivery"]:::test
        TRC["TestRedisClient"]:::test
    end

    EC("ErrorChannel"):::error

    Redis[(Redis Cluster)]:::db

    CP -->|Publish| QM
    QM -->|Enqueue Ready| Redis
    Redis -->|Prefetch Goroutine| DE
    DE -->|Deliver| CC
    DE -->|DeliverBatch| CBC
    CC -->|Ack/Reject/Push| QM
    Cleaner -->|Recover Expired| Redis
    CM -->|Heartbeat| Redis
    SC -->|CollectStats| Redis
    HM -->|Wrap/Unwrap| QM

    CM --> EC
    QM --> EC
    DE --> EC

    TC -.->|Mock Connection| CM
    TQ -.->|Mock Queue| QM
    TD -.->|Mock Delivery| DE
    TRC -.->|Mock Redis Client| CM

    click CM "https://github.com/wellle/rmq/blob/master/connection.go"
    click QM "https://github.com/wellle/rmq/blob/master/queue.go"
    click DE "https://github.com/wellle/rmq/blob/master/delivery.go"
    click Cleaner "https://github.com/wellle/rmq/blob/master/cleaner.go"
    click SC "https://github.com/wellle/rmq/blob/master/stats.go"
    click HM "https://github.com/wellle/rmq/blob/master/header.go"
    click CP "https://github.com/wellle/rmq/blob/master/example/producer/main.go"
    click CC "https://github.com/wellle/rmq/blob/master/example/consumer/main.go"
    click CBC "https://github.com/wellle/rmq/blob/master/example/batch_consumer/main.go"
    click TC "https://github.com/wellle/rmq/blob/master/test_connection.go"
    click TQ "https://github.com/wellle/rmq/blob/master/test_queue.go"
    click TD "https://github.com/wellle/rmq/blob/master/test_delivery.go"
    click TRC "https://github.com/wellle/rmq/blob/master/test_redis_client.go"
    click UF "https://github.com/wellle/rmq/blob/master/errors.go"
    click SH "https://github.com/wellle/rmq/blob/master/state.go"
    click CR "https://github.com/wellle/rmq/blob/master/example/returner/main.go"
    click CPg "https://github.com/wellle/rmq/blob/master/example/purger/main.go"
    click CH "https://github.com/wellle/rmq/blob/master/example/handler/main.go"

    classDef client fill:#a2f4c7,stroke:#333,stroke-width:1px
    classDef subsystem fill:#9dc3e6,stroke:#333,stroke-width:1px
    classDef db fill:#f4b183,stroke:#333,stroke-width:1px
    classDef test fill:#d9d2e9,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef error fill:#ffe599,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
