# stym06/rebuf architecture

> Architecture diagram of the GitHub repository stym06/rebuf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stym06/rebuf
- Repository: https://github.com/stym06/rebuf

## Overview

##### What this project is (type + purpose)
**Project type:** Go library (not a service) implementing a **segmented Write-Ahead Log (WAL)**.

**Purpose:** Provide a **durable buffer** on local disk for bytes/events during downstream outages, then **replay** later (optionally from a specific segment). It’s used as an embedded component inside other applications (outbox pattern, webhook retries, IoT buffering, etc.).

- Pure Go stdlib, **single-process embedded module**
- **Segmented files** + rotation/retention
- **CRC32 integrity** per entry
- **Sync strategies** (fsync every write / periodic / manual)
- **Replay + Purge** lifecycle
- Concurrency protection via **mutex** (important: single shared state)

##### What the file tree implies about structure
From the tree:

- `rebuf/`
- `rebuf.go`, `rebuf_test.go`
- `utils/`
- `utils.go`, `utils_test.go`
Supporting helpers (likely file ops, encoding/decoding, segment naming, etc.).
- `example.go`
Demonstrates how a **consumer application** embeds/uses `rebuf`.
- `.github/workflows/go.yml`, `Makefile`, `go.mod`

##### README insights to encode
From README:
- Public API: `New`, `Write`, `WriteBatch`, `Replay`, `ReplayFrom`, `Purge`, `PurgeThrough`, `Sync`, `Close`, metrics helpers.
- On-disk format: `[size][crc32][data]`, segment names `rebuf-0`, `rebuf-1`, … and active `rebuf.tmp`.
- Segment rotation: bounded by `WithMaxLogSize`, retention by `WithMaxSegments`.
- Sync: `WithFsyncTime`, `WithSyncStrategy` including `SyncEveryWrite`.

These are not “code details”; they are **architectural mechanics** and should be visualized as **subcomponents and flows**.

##### Host Application (User / Producer / Consumer)
- This is whatever system uses `rebuf` (message producer, webhook dispatcher, DB-outbox worker, etc.).
- It calling `Write*` and `Replay*`.

##### rebuf Library (Core)
- **API Layer**
- Methods: `Write`, `WriteBatch`, `Replay`, `ReplayFrom`, `Purge*`, `Sync`, `Close`
- **Concurrency Guard**
- **Segment Manager**
- Segment rotation (max log size)
- Segment retention (max segments)
- Segment naming (`rebuf-N`, active `rebuf.tmp`)
- **Record Codec**
- Encodes/decodes `[size][crc32][data]`
- CRC32 compute + verify
- **Sync Manager**
- Sync strategy (every write vs periodic vs manual)
- Timer-based fsync (if configured)
- **Replay Engine**
- Sequential read of segments, verify CRC, invoke callback
- Supports “start from segmentId”

##### Filesystem / Disk (Persistent Storage)
- Directory path (`Dir()`) and files:
- `rebuf.tmp` (active)
- `rebuf-0..N` (completed segments)

- The “service that might be down” (DB, HTTP endpoint, message broker, etc.).
- Replay callback is what pushes data into this service.

Flows explicitly:

**Write path (durable buffering):**
- Host App → `rebuf.Write / WriteBatch`
- Inside rebuf:
- API → Concurrency Guard (mutex)
- → Record Codec (size + crc)
- → Filesystem write to `rebuf.tmp`
- → Sync Manager decides fsync behavior (every write / periodic / manual)
- → Segment Manager may rotate `rebuf.tmp` → `rebuf-K` when max size reached
- Segment retention may delete oldest segments when exceeding max segments

**Replay path (recovery/drain):**
- Host App → `rebuf.Replay / ReplayFrom`
- Inside rebuf:
- API → Concurrency Guard
- → Replay Engine reads `rebuf-0..N` (and possibly `rebuf.tmp` depending on implementation—if unknown, annotate “active segment may be included”)
- → Record Codec validates CRC32
- → Callback invocation
- Callback → Downstream Service (HTTP/DB/broker)
- After success, Host App → `rebuf.Purge / PurgeThrough` → Filesystem delete segments

**Operational / lifecycle calls:**
- Host App → `rebuf.Sync` (manual flush)
- Host App → `rebuf.Close` → flush + sync + close file handles

##### Architectural patterns and design principles
- **WAL / Durable Buffer** pattern (embedded)
- **Segmented log** with rotation + retention
- **Integrity checking** via CRC32 on replay
- **Callback-based replay** (push model into downstream)
- **Single shared state with mutex** (thread-safe usage)
- **Configurable durability vs throughput** via sync strategy

##### Relevant technologies / key decisions
Annotate:
- Language: **Go**
- Dependencies: **stdlib only**
- Storage: **local filesystem**, append-only segments
- Checksum: **CRC32 IEEE**
- Sync: **fsync**, periodic timer vs per-write

- Host Application uses `rebuf` to persist data on disk and replay to downstream service.
- Expand `rebuf` into Segment Manager / Sync Manager / Replay Engine / Codec / Mutex.
- Write + rotate
- Replay + purge

This gives principal-engineer-level clarity without pretending it’s a distributed architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =======================
  %% Context (embedded library)
  %% =======================
  subgraph "Context (Embedded Durable Buffer / WAL)"
    direction TB

    Host["Host Application\n(Producer/Consumer)"]:::host
    Example["Integration Example\n(consumer embedding rebuf)"]:::host

    subgraph "rebuf (Go WAL Library - single-process, stdlib-only)"
      direction TB

      API["Public API Layer\nNew,Write,WriteBatch,Replay,ReplayFrom,\nPurge,PurgeThrough,Sync,Close"]:::api
      Mutex["Concurrency Guard\nSingle shared state + Mutex\n(thread-safe)"]:::guard

      subgraph "Core Mechanics"
        direction TB
        Codec["Record Codec\nFrame: [uint64 size][crc32][bytes]\nencode/decode"]:::codec
        CRC["CRC32 Integrity\nCompute + Verify (IEEE)"]:::codec
        SegMgr["Segment Manager\nActive: rebuf.tmp\nCompleted: rebuf-0..N\nRotate(max log size)\nRetain(max segments)"]:::segment
        SyncMgr["Sync Manager\nSyncEveryWrite | Periodic(fsync time) | Manual\nfsync() decisions"]:::sync
        Replay["Replay Engine\nSequential segment read\nOptional start segmentId\nCallback-based replay"]:::replay
      end

      Logger["Logger (slog)\n(optional dependency)"]:::external
    end

    Disk[("Filesystem / Local Disk\nlogDir (Dir())\nrebuf.tmp + rebuf-0..N")]:::disk
    Downstream["Downstream Service\n(DB/HTTP/Broker)\n(may be unavailable)"]:::external
  end

  %% =======================
  %% Write / Buffer Flow
  %% =======================
  Host -->|"Write(entry bytes) / WriteBatch"| API
  Example -->|"Demonstrates usage"| API

  API -->|"locks"| Mutex
  Mutex -->|"encode frame"| Codec
  Codec -->|"compute crc32"| CRC
  CRC -->|"append [size][crc][data]"| Disk
  Mutex -->|"sync policy"| SyncMgr
  SyncMgr -->|"fsync() now/periodic/manual"| Disk
  Mutex -->|"rotate/retain"| SegMgr
  SegMgr -->|"rename rebuf.tmp -> rebuf-K\ncreate new rebuf.tmp"| Disk
  SegMgr -->|"delete oldest segments\n(when max segments exceeded)"| Disk

  %% =======================
  %% Replay / Drain Flow
  %% =======================
  Host -->|"Replay(callback) / ReplayFrom(segmentId)"| API
  Mutex -->|"read segments rebuf-0..N\n(active rebuf.tmp may be included)"| Replay
  Disk -->|"sequential read"| Replay
  Replay -->|"decode frame"| Codec
  Codec -->|"verify crc32"| CRC
  Replay -->|"invoke callback(bytes)"| Host
  Host -->|"deliver events"| Downstream

  %% Purge lifecycle after successful drain
  Host -->|"Purge() / PurgeThrough(segmentId)"| API
  Mutex -->|"delete acknowledged segments"| SegMgr
  SegMgr -->|"unlink segment files"| Disk

  %% Operational / lifecycle calls
  Host -->|"Sync() (manual flush)"| API
  Host -->|"Close() (flush+sync+close handles)"| API

  %% Optional logging
  API -.->|"logs/diagnostics"| Logger

  %% =======================
  %% Delivery / Repo Artifacts (optional)
  %% =======================
  subgraph "Repo / Delivery"
    direction TB
    Readme["README\nAPI + On-disk format"]:::doc
    GoMod["go.mod\nmodule metadata"]:::meta
    Make["Makefile\nbuild/dev commands"]:::meta
    CI["GitHub Actions CI\nGo workflow"]:::ci
    TestsCore["Core Tests\n(WAL behavior)"]:::test
    TestsUtils["Utils Tests"]:::test
    Utils["Utilities / Helpers\n(file ops, encoding, segment helpers)"]:::util
  end

  Utils -->|"supports"| SegMgr
  Utils -->|"supports"| Codec
  TestsCore -->|"validates"| API
  TestsUtils -->|"validates"| Utils
  CI -->|"runs tests/build"| TestsCore
  CI -->|"runs tests/build"| TestsUtils
  Readme -->|"documents usage"| Host
  GoMod -->|"defines module"| CI
  Make -->|"invokes"| CI

  %% =======================
  %% Click Events (paths only)
  %% =======================
  click API "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click Mutex "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click SegMgr "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click Codec "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click CRC "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click SyncMgr "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click Replay "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf.go"
  click Utils "https://github.com/stym06/rebuf/blob/main/utils/utils.go"
  click Example "https://github.com/stym06/rebuf/blob/main/example.go"
  click Readme "https://github.com/stym06/rebuf/blob/main/README.md"
  click GoMod "https://github.com/stym06/rebuf/blob/main/go.mod"
  click CI "https://github.com/stym06/rebuf/blob/main/.github/workflows/go.yml"
  click Make "https://github.com/stym06/rebuf/tree/main/Makefile"
  click TestsCore "https://github.com/stym06/rebuf/blob/main/rebuf/rebuf_test.go"
  click TestsUtils "https://github.com/stym06/rebuf/blob/main/utils/utils_test.go"

  %% =======================
  %% Styles (color)
  %% =======================
  classDef host fill:#E6F3FF,stroke:#2B6CB0,stroke-width:1.5px,color:#0B2A4A
  classDef api fill:#DFF7E6,stroke:#2F855A,stroke-width:1.5px,color:#12361F
  classDef guard fill:#FFF4D6,stroke:#B7791F,stroke-width:1.5px,color:#4A2C0A
  classDef segment fill:#FDE2E2,stroke:#C53030,stroke-width:1.5px,color:#4A0B0B
  classDef codec fill:#EFE7FF,stroke:#6B46C1,stroke-width:1.5px,color:#24105A
  classDef sync fill:#E8FFF7,stroke:#0F766E,stroke-width:1.5px,color:#083C39
  classDef replay fill:#FFE8F1,stroke:#B83280,stroke-width:1.5px,color:#3A0A23
  classDef disk fill:#FFF1F1,stroke:#9B2C2C,stroke-width:2px,color:#4A0B0B
  classDef external fill:#F3F4F6,stroke:#4B5563,stroke-dasharray: 6 4,stroke-width:1.5px,color:#111827
  classDef util fill:#E9FFEA,stroke:#2F855A,stroke-width:1.5px,color:#12361F
  classDef doc fill:#FFFBEA,stroke:#B7791F,stroke-width:1.5px,color:#4A2C0A
  classDef meta fill:#EEF2FF,stroke:#4338CA,stroke-width:1.5px,color:#111827
  classDef ci fill:#E0F2FE,stroke:#0369A1,stroke-width:1.5px,color:#082F49
  classDef test fill:#FAF5FF,stroke:#7C3AED,stroke-width:1.5px,color:#2E1065
```
