How pkritiotis/go-outbox works
Project type and purpose
– This is a Go library + example daemon implementing the Transactional Outbox Pattern.
– It provides two main use-cases:
• A “publisher” module you embed in your application to enqueue messages inside your SQL transaction.
• A standalone “dispatcher” process that polls the outbox table, publishes enqueued messages to a broker, and manages retries/cleanup.
File structure
– Top-level Go interfaces and entrypoints:
• broker.go / broker_mock.go: broker interface for sending messages.
• store.go / store_mock.go: data-store interface for persisting Outbox rows.
• publisher.go + dispatcher.go: core orchestrators for “enqueue” vs. “flush” workflows.
• record_processor.go, record_cleaner.go, record_unlocker.go: sub-components of the dispatcher.
– Vendor directory holds dependencies; internal/time and internal/uuid provide testable hooks.
– broker/kafka implements KafkaBroker; store/mysql implements MySQL store.
– examples/mySQL-Kafka shows end-to-end wiring (MySQL ↔ Outbox lib ↔ Kafka).
– CI workflows, Makefile, docs all point to a production-grade library.
Insights from the README
– The library supports: transactional enqueue, row locking, retry limits, lock cleanup, message retention.
– Pluggable broker interface (Kafka) and store interface (MySQL).
– Sample code shows how an application uses repo.Add within sql.Tx, then how the dispatcher polls and pushes to Kafka.
a. Main components (group into layers)
• APPLICATION (your service)
• OUTBOX LIBRARY – Publisher API
• DATABASE / Outbox table (MySQL)
• DISPATCHER SERVICE – includes:
– RecordFetcher & Locker
– RecordProcessor (sends to Broker)
– RecordUnlocker (releases stale locks)
– CleanupWorker (deletes old records)
• MESSAGE BROKER (Kafka)
1. App → Publisher.Add() → sql.Tx → Outbox row INSERT
2. Commit → row becomes visible to dispatcher
3. Dispatcher (periodic) → SELECT unprocessed rows
4. Dispatcher → UPDATE lock metadata on selected rows
5. Dispatcher → RecordProcessor → Broker.Publish(topic, data) → Kafka
6. On success → Dispatcher → UPDATE row state to “processed” + timestamps
7. RecordUnlocker → UNLOCK rows with expired locks (if broker hung)
8. CleanupWorker → DELETE rows older than retention window
c. Architectural patterns & design principles
– Transactional Outbox: atomic write + deferred publish
– Locking pattern for concurrency control
– Modular design: interfaces + pluggable implementations
– Background worker pattern for periodic tasks
– Dependency inversion: broker and store abstractions
d. Tech stack & libraries
– Go modules (internal/time, internal/uuid for determinism in tests)
– MySQL driver and SQL Tx
– Sarama Kafka client (in broker/kafka)
– go-outbox core packages: publisher, dispatcher, record_*.go