# pkritiotis/go-outbox architecture

> Architecture diagram of the GitHub repository pkritiotis/go-outbox, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pkritiotis/go-outbox
- Repository: https://github.com/pkritiotis/go-outbox

## Overview

##### Project type and purpose
– This is a Go library + example daemon implementing the Transactional Outbox Pattern.
– It provides two main use-cases:
- A “publisher” module you embed in your application to enqueue messages inside your SQL transaction.
- A standalone “dispatcher” process that polls the outbox table, publishes enqueued messages to a broker, and manages retries/cleanup.

##### File structure
– Top-level Go interfaces and entrypoints:
- broker.go / broker_mock.go: broker interface for sending messages.
- store.go / store_mock.go: data-store interface for persisting Outbox rows.
- publisher.go + dispatcher.go: core orchestrators for “enqueue” vs. “flush” workflows.
- record_processor.go, record_cleaner.go, record_unlocker.go: sub-components of the dispatcher.
– Vendor directory holds dependencies; internal/time and internal/uuid provide testable hooks.
– broker/kafka implements KafkaBroker; store/mysql implements MySQL store.
– examples/mySQL-Kafka shows end-to-end wiring (MySQL ↔ Outbox lib ↔ Kafka).
– CI workflows, Makefile, docs all point to a production-grade library.

##### Insights from the README
– The library supports: transactional enqueue, row locking, retry limits, lock cleanup, message retention.
– Pluggable broker interface (Kafka) and store interface (MySQL).
– Sample code shows how an application uses repo.Add within sql.Tx, then how the dispatcher polls and pushes to Kafka.

a. Main components (group into layers)
- APPLICATION (your service)
- OUTBOX LIBRARY – Publisher API
- DATABASE / Outbox table (MySQL)
- DISPATCHER SERVICE – includes:
– RecordFetcher & Locker
– RecordProcessor (sends to Broker)
– RecordUnlocker (releases stale locks)
– CleanupWorker (deletes old records)
- MESSAGE BROKER (Kafka)
1. App → Publisher.Add() → sql.Tx → Outbox row INSERT
2. Commit → row becomes visible to dispatcher
3. Dispatcher (periodic) → SELECT unprocessed rows
4. Dispatcher → UPDATE lock metadata on selected rows
5. Dispatcher → RecordProcessor → Broker.Publish(topic, data) → Kafka
6. On success → Dispatcher → UPDATE row state to “processed” + timestamps
7. RecordUnlocker → UNLOCK rows with expired locks (if broker hung)
8. CleanupWorker → DELETE rows older than retention window
c. Architectural patterns & design principles
– Transactional Outbox: atomic write + deferred publish
– Locking pattern for concurrency control
– Modular design: interfaces + pluggable implementations
– Background worker pattern for periodic tasks
– Dependency inversion: broker and store abstractions
d. Tech stack & libraries
– Go modules (internal/time, internal/uuid for determinism in tests)
– MySQL driver and SQL Tx
– Sarama Kafka client (in broker/kafka)
– go-outbox core packages: publisher, dispatcher, record_*.go

## Diagram (Mermaid)

```mermaid
graph TB
  subgraph "Application Layer"
    App["Your Application"]:::producer
    Publisher["Publisher Module"]:::producer
    Publisher -->|"calls Add()"| App
    PublisherTest["Publisher Tests"]:::test
  end

  subgraph "Core Interfaces"
    BrokerIface["Broker Interface"]:::interface
    StoreIface["Store Interface"]:::interface
  end

  subgraph "Outbox Database"
    DB["MySQL Outbox Table"]:::db
  end

  subgraph "Dispatcher Service"
    Dispatcher["Dispatcher Service"]:::dispatcher
    subgraph "Dispatch Workers"
      Fetcher["RecordFetcher & Locker"]:::dispatcher
      Processor["RecordProcessor"]:::dispatcher
      Unlocker["RecordUnlocker"]:::cleanup
      Cleaner["CleanupWorker"]:::cleanup
    end
  end

  subgraph "Brokers"
    KafkaBroker["Kafka Broker Impl"]:::external
  end

  subgraph "Stores"
    MySQLStore["MySQL Store Impl"]:::external
  end

  subgraph "Internal Test Hooks"
    TimeHook["Time Hook"]:::internal
    UUIDHook["UUID Hook"]:::internal
  end

  subgraph "Test Mocks & Tests"
    BrokerMock["Broker Mock"]:::test
    StoreMock["Store Mock"]:::test
    TimeMock["Time Mock"]:::test
    UUIDMock["UUID Mock"]:::test
    DispatcherTest["Dispatcher Tests"]:::test
    ProcessorTest["RecordProcessor Tests"]:::test
    UnlockerTest["RecordUnlocker Tests"]:::test
    CleanerTest["RecordCleaner Tests"]:::test
  end

  subgraph "Examples"
    ExampleApp["Example App"]:::example
    ExampleSchema["Example Schema"]:::example
  end

  App -->|"uses"| Publisher
  Publisher -->|"INSERT Outbox Row"| DB
  Publisher -->|"implements"| StoreIface
  Publisher -->|"implements"| BrokerIface

  Dispatcher -->|"periodic"| Fetcher
  Fetcher -->|"SELECT & LOCK rows"| DB
  Fetcher -->|"passes rows"| Processor
  Processor -->|"Broker.Publish()"| KafkaBroker
  Processor -->|"UPDATE row state"| DB
  Unlocker -->|"UNLOCK expired rows"| DB
  Cleaner -->|"DELETE old rows"| DB
  Dispatcher -->|"uses"| StoreIface
  Processor -->|"implements"| BrokerIface

  KafkaBroker -->|"writes to"| KafkaBroker
  MySQLStore -->|"reads/writes"| DB

  ExampleApp -->|"wires"| MySQLStore
  ExampleApp -->|"wires"| KafkaBroker
  ExampleSchema -->|"defines"| DB

  classDef producer fill:#e0f7e9,stroke:#2e7d32,color:#2e7d32;
  classDef dispatcher fill:#e3f2fd,stroke:#1976d2,color:#1976d2;
  classDef cleanup fill:#f5f5f5,stroke:#9e9e9e,color:#616161;
  classDef external fill:#fff3e0,stroke:#ef6c00,color:#e65100;
  classDef db fill:#efebe9,stroke:#5d4037,color:#3e2723,shape:cylinder;
  classDef interface fill:#e8eaf6,stroke:#3949ab,color:#1a237e,stroke-dasharray: 5 5;
  classDef internal fill:#f3e5f5,stroke:#8e24aa,color:#4a148c,stroke-dasharray: 2 2;
  classDef test fill:#ffebee,stroke:#c62828,color:#b71c1c;
  classDef example fill:#e1f5fe,stroke:#0288d1,color:#01579b;

  click Publisher "https://github.com/pkritiotis/go-outbox/blob/main/publisher.go"
  click Dispatcher "https://github.com/pkritiotis/go-outbox/blob/main/dispatcher.go"
  click Processor "https://github.com/pkritiotis/go-outbox/blob/main/record_processor.go"
  click Unlocker "https://github.com/pkritiotis/go-outbox/blob/main/record_unlocker.go"
  click Cleaner "https://github.com/pkritiotis/go-outbox/blob/main/record_cleaner.go"
  click BrokerIface "https://github.com/pkritiotis/go-outbox/blob/main/broker.go"
  click KafkaBroker "https://github.com/pkritiotis/go-outbox/blob/main/broker/kafka/kafka.go"
  click StoreIface "https://github.com/pkritiotis/go-outbox/blob/main/store.go"
  click MySQLStore "https://github.com/pkritiotis/go-outbox/blob/main/store/mysql/mysql.go"
  click TimeHook "https://github.com/pkritiotis/go-outbox/blob/main/internal/time/time.go"
  click UUIDHook "https://github.com/pkritiotis/go-outbox/blob/main/internal/uuid/uuid.go"
  click PublisherTest "https://github.com/pkritiotis/go-outbox/blob/main/publisher_test.go"
  click DispatcherTest "https://github.com/pkritiotis/go-outbox/blob/main/dispatcher_test.go"
  click ProcessorTest "https://github.com/pkritiotis/go-outbox/blob/main/record_processor_test.go"
  click CleanerTest "https://github.com/pkritiotis/go-outbox/blob/main/record_cleaner_test.go"
  click UnlockerTest "https://github.com/pkritiotis/go-outbox/blob/main/record_unlocker_test.go"
  click BrokerMock "https://github.com/pkritiotis/go-outbox/blob/main/broker_mock.go"
  click StoreMock "https://github.com/pkritiotis/go-outbox/blob/main/store_mock.go"
  click TimeMock "https://github.com/pkritiotis/go-outbox/blob/main/internal/time/mock_time.go"
  click UUIDMock "https://github.com/pkritiotis/go-outbox/blob/main/internal/uuid/mock_uuid.go"
  click ExampleApp "https://github.com/pkritiotis/go-outbox/blob/main/examples/mySQL-Kafka/main.go"
  click ExampleSchema "https://github.com/pkritiotis/go-outbox/blob/main/examples/mySQL-Kafka/schema/outbox.sql"
```
