How stym06/rebuf works
What this project is (type + purpose)
Project type: Go library (not a service) implementing a segmented Write-Ahead Log (WAL).
Purpose: Provide a durable buffer on local disk for bytes/events during downstream outages, then replay later (optionally from a specific segment). It’s used as an embedded component inside other applications (outbox pattern, webhook retries, IoT buffering, etc.).
• Pure Go stdlib, single-process embedded module
• Segmented files + rotation/retention
• CRC32 integrity per entry
• Sync strategies (fsync every write / periodic / manual)
• Replay + Purge lifecycle
• Concurrency protection via mutex (important: single shared state)
What the file tree implies about structure
From the tree:
• rebuf/
• rebuf.go, rebuf_test.go
• utils/
• utils.go, utils_test.go
Supporting helpers (likely file ops, encoding/decoding, segment naming, etc.).
• example.go
Demonstrates how a consumer application embeds/uses rebuf.
• .github/workflows/go.yml, Makefile, go.mod
README insights to encode
From README:
• Public API: New, Write, WriteBatch, Replay, ReplayFrom, Purge, PurgeThrough, Sync, Close, metrics helpers.
• On-disk format: [size][crc32][data], segment names rebuf-0, rebuf-1, … and active rebuf.tmp.
• Segment rotation: bounded by WithMaxLogSize, retention by WithMaxSegments.
• Sync: WithFsyncTime, WithSyncStrategy including SyncEveryWrite.
These are not “code details”; they are architectural mechanics and should be visualized as subcomponents and flows.
Host Application (User / Producer / Consumer)
• This is whatever system uses rebuf (message producer, webhook dispatcher, DB-outbox worker, etc.).
• It calling Write* and Replay*.
rebuf Library (Core)
• API Layer
• Methods: Write, WriteBatch, Replay, ReplayFrom, Purge*, Sync, Close
• Concurrency Guard
• Segment Manager
• Segment rotation (max log size)
• Segment retention (max segments)
• Segment naming (rebuf-N, active rebuf.tmp)
• Record Codec
• Encodes/decodes [size][crc32][data]
• CRC32 compute + verify
• Sync Manager
• Sync strategy (every write vs periodic vs manual)
• Timer-based fsync (if configured)
• Replay Engine
• Sequential read of segments, verify CRC, invoke callback
• Supports “start from segmentId”
Filesystem / Disk (Persistent Storage)
• Directory path (Dir()) and files:
• rebuf.tmp (active)
• rebuf-0..N (completed segments)
• The “service that might be down” (DB, HTTP endpoint, message broker, etc.).
• Replay callback is what pushes data into this service.
Flows explicitly:
Write path (durable buffering):
• Host App → rebuf.Write / WriteBatch
• Inside rebuf:
• API → Concurrency Guard (mutex)
• → Record Codec (size + crc)
• → Filesystem write to rebuf.tmp
• → Sync Manager decides fsync behavior (every write / periodic / manual)
• → Segment Manager may rotate rebuf.tmp → rebuf-K when max size reached
• Segment retention may delete oldest segments when exceeding max segments
Replay path (recovery/drain):
• Host App → rebuf.Replay / ReplayFrom
• Inside rebuf:
• API → Concurrency Guard
• → Replay Engine reads rebuf-0..N (and possibly rebuf.tmp depending on implementation—if unknown, annotate “active segment may be included”)
• → Record Codec validates CRC32
• → Callback invocation
• Callback → Downstream Service (HTTP/DB/broker)
• After success, Host App → rebuf.Purge / PurgeThrough → Filesystem delete segments
Operational / lifecycle calls:
• Host App → rebuf.Sync (manual flush)
• Host App → rebuf.Close → flush + sync + close file handles
Architectural patterns and design principles
• WAL / Durable Buffer pattern (embedded)
• Segmented log with rotation + retention
• Integrity checking via CRC32 on replay
• Callback-based replay (push model into downstream)
• Single shared state with mutex (thread-safe usage)
• Configurable durability vs throughput via sync strategy
Relevant technologies / key decisions
Annotate:
• Language: Go
• Dependencies: stdlib only
• Storage: local filesystem, append-only segments
• Checksum: CRC32 IEEE
• Sync: fsync, periodic timer vs per-write
• Host Application uses rebuf to persist data on disk and replay to downstream service.
• Expand rebuf into Segment Manager / Sync Manager / Replay Engine / Codec / Mutex.
• Write + rotate
• Replay + purge
This gives principal-engineer-level clarity without pretending it’s a distributed architecture.