# adityagaddhyan/bbolt architecture

> Architecture diagram of the GitHub repository adityagaddhyan/bbolt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adityagaddhyan/bbolt
- Repository: https://github.com/adityagaddhyan/bbolt

## Overview

This repository is the Go implementation of an embedded, single-file key/value database: bbolt. The architecture is centered on a public database API at the repository root, a set of internal storage-engine subsystems, platform-specific OS bindings, and a separate command-line tool built on top of the same library.

#### High-level architecture

The core design is a memory-mapped B+tree database with ACID transactions and MVCC behavior:

- The database lives in one file on disk.
- The file is memory-mapped into the process.
- Reads usually traverse the mapped pages directly.
- Writes happen in a single writable transaction at a time.
- Commit is copy-on-write: dirty pages are written out, then a new meta page is atomically promoted.

This creates a clear boundary between:
- the public API used by applications,
- the storage engine internals that manage pages, nodes, freelists, and transactions,
- OS-specific file/memory primitives,
- and tooling for inspection, repair, benchmarking, and maintenance.

#### Main subsystems

##### 1) Public database API at the repository root

The root package is the library consumed by application code. The main types and responsibilities are:

- `DB`: opens/closes the database file, manages locking, mmap, and transaction lifecycle.
- `Tx`: represents a read-only or read-write transaction.
- `Bucket`: provides namespaced key/value access and nested buckets.
- `Cursor`: performs ordered traversal over keys and buckets.
- Supporting APIs for stats, backup, compaction, batching, and validation.

The README examples reflect the intended usage pattern:
- `Open()` creates or opens the file and maps it into memory.
- `View()` runs a read-only transaction.
- `Update()` runs a writable transaction.
- `Batch()` groups multiple write requests.
- `WriteTo()` and `CopyFile()` support hot backup.

Architecturally, this is the stable API boundary. Everything else in the repository exists to make these operations correct, durable, and performant.

##### 2) Transaction and commit engine

The transaction subsystem is the heart of the database.

Data flow:
1. `DB.Begin()` creates a transaction.
2. Read transactions get a consistent snapshot of the mmap’d file.
3. Write transactions create in-memory dirty state by materializing pages into nodes.
4. `Tx.Commit()` flushes dirty pages and advances the database generation via a new meta page.
5. `Tx.Rollback()` discards in-memory state.

Important architectural properties:
- Only one writable transaction may exist at a time.
- Multiple readers are allowed concurrently.
- A read-write transaction cannot safely finalize while read-only transactions still pin old mappings/pages.
- Commit uses a two-phase write pattern:
  - write dirty data pages and fsync,
  - then write the new meta page and fsync again.
- The meta page checksum protects against partially written or torn writes.

This transaction engine is the main consistency boundary of the system.

##### 3) B+tree storage model: bucket, cursor, node, and page layers

bbolt stores data in a B+tree-like structure. The repository divides the tree logic across several concepts:

- `page`: on-disk physical page layout.
- `node`: in-memory mutable representation used during write transactions.
- `bucket`: logical namespace that owns key/value pairs and nested buckets.
- `cursor`: traversal mechanism that walks the tree, hiding page/node navigation.
- `inode` and other common structures: internal record representations inside pages and nodes.

Data flow through this layer looks like:
- A `Bucket.Put()` or `Bucket.Get()` starts at the bucket root.
- A cursor traverses the tree to locate the leaf page or node.
- For reads, values may be returned directly from the mmap’d file when safe.
- For writes, pages are materialized into mutable nodes.
- Changes are then propagated upward as needed before commit.

The logical boundary here is important:
- Buckets provide the user-facing data model.
- Cursors provide ordered access.
- Nodes and pages are internal representations of the same tree at different phases of the lifecycle.

##### 4) Free space management

The freelist subsystem tracks unused pages so the database can reuse space without truncating the file.

Repository signals:
- `internal/freelist/array.go`
- `internal/freelist/hashmap.go`
- `internal/freelist/shared.go`

This indicates more than one freelist implementation strategy, likely selected based on runtime/build constraints or performance tradeoffs.

Architecture role:
- When transactions free pages, those pages are added to the freelist.
- Future commits allocate from the freelist before growing the file.
- This is essential to the copy-on-write model because old pages cannot be immediately reclaimed if readers still reference them.

The freelist is a major persistence boundary because it mediates file growth and reuse.

##### 5) Platform abstraction layer

The repository has OS-specific files such as:
- `bolt_linux.go`
- `bolt_windows.go`
- `bolt_unix.go`
- `bolt_aix.go`
- `bolt_android.go`
- `bolt_openbsd.go`
- `bolt_solaris.go`
- `mlock_unix.go`
- `mlock_windows.go`
- `boltsync_unix.go`

These isolate platform-dependent behavior such as:
- file locking,
- mmap setup/teardown,
- memory locking,
- sync/fsync semantics,
- and platform-specific edge cases.

Architecturally, this layer is the boundary between the storage engine and the host operating system. The core database logic depends on these abstractions, but not on any one OS implementation.

##### 6) `internal/common`: shared low-level types and page layout

`internal/common` holds cross-cutting, architecture-sensitive definitions:
- `meta.go`
- `page.go`
- `inode.go`
- `bucket.go`
- `types.go`
- `verify.go`
- per-architecture files like `bolt_amd64.go`, `bolt_arm64.go`, etc.

This package exists to centralize binary layout and unsafe operations:
- page headers,
- meta page format,
- bucket/node record layout,
- endian and alignment concerns,
- architecture-specific constants and helpers.

Because the file format is fixed and memory-mapped, these low-level types are part of the on-disk contract. They are a critical boundary between portable Go code and the serialized database format.

##### 7) Validation and integrity checking

There are dedicated validation paths:
- `tx_check.go`
- `internal/common/verify.go`
- tests like `tx_check_test.go`

These are used to verify internal invariants and detect corruption or structural problems in the database tree, freelist, and meta information.

This subsystem acts as a diagnostic boundary:
- normal API operations are optimized for use,
- validation code is used for correctness checking, testing, and repair tooling.

##### 8) Command-line tooling in `cmd/bbolt`

The CLI is a separate consumer of the library, located under `cmd/bbolt`. It is not the core database engine; it is an operational front-end for inspecting and manipulating database files.

Subcommands in `cmd/bbolt/command` indicate the CLI’s role:
- inspect and info commands: report database structure and metadata,
- dump/get/keys/buckets: view logical contents,
- page/pages/page-item: inspect physical page structures,
- stats: show internal statistics,
- check: validate consistency,
- compact: rewrite the database to a smaller or cleaner file,
- surgery/meta/freelist: low-level repair and manipulation operations,
- bench/version: performance and version reporting.

There is also `internal/surgeon` and `internal/guts_cli`, suggesting the CLI has specialized plumbing for deep inspection and “surgical” maintenance of database internals.

Architecturally, the CLI crosses both the public API boundary and the internal storage boundary:
- ordinary commands use the public bucket/tx APIs,
- repair/inspection commands may reach into file-format internals.

##### 9) Testing, robustness, and fault injection

This repository has an unusually strong test and resilience layer:
- unit tests around buckets, cursors, nodes, transactions, freelist behavior, and allocation,
- white-box tests for internal state,
- concurrency and multi-DB tests,
- simulation tests,
- failpoint tests,
- robustness tests,
- platform-specific tests,
- `tests/dmflakey` for device-mapper / flaky I/O scenarios,
- `tests/robustness` for power-failure style behavior.

These tests reflect the key architectural risks of the system:
- concurrent access,
- crash consistency,
- page reuse correctness,
- file system behavior,
- cross-platform differences,
- and corruption resistance.

#### Primary data flows

##### Read path
1. Application opens the database.
2. DB acquires the necessary file lock and memory-maps the file.
3. A read-only transaction snapshots the current meta state.
4. A bucket/cursor walks the B+tree.
5. Values are returned directly from mapped memory when possible.
6. Transaction ends without modifying disk state.

This path is optimized for low allocation and fast range scans.

##### Write path
1. Application begins a writable transaction.
2. Mutations are applied to in-memory nodes, not directly to disk pages.
3. Page allocations are taken from the freelist or by extending the file.
4. On commit, dirty pages are serialized and written to disk.
5. Fsync occurs.
6. A new meta page is written and synced, making the transaction durable and visible.
7. Old pages remain available until no readers reference them.

This path preserves crash safety through copy-on-write and meta-page switching.

##### Backup/compaction path
- Read-only transactions can stream a consistent snapshot to a writer via `WriteTo()`.
- `CopyFile()` and the `compact` CLI command provide file-level rewriting.
- These operations use the same consistency guarantees as normal reads.

#### Important architectural boundaries

##### Public API vs storage internals
The root package exposes a small stable API. Internal packages hold the file format, page structures, freelist algorithms, and traversal logic. This separation is intentional: applications should depend on the API, not on physical storage details.

##### Logical data model vs physical file model
Buckets, keys, and cursors are the logical abstraction. Pages, nodes, meta pages, and freelists are the physical implementation. The CLI’s page/surgery commands are where this boundary becomes visible.

##### Portable logic vs platform-specific behavior
Cross-platform files and `internal/common` hide OS and CPU differences. This is crucial because the database file format is memory-mapped and architecture-sensitive.

##### Normal operation vs diagnostic/repair tooling
`cmd/bbolt`, `internal/surgeon`, and the validation code provide a separate operational plane for inspection, stats, verification, and repair. They are not required for normal embedded usage but are essential for maintenance and debugging.

#### What an architecture diagram should emphasize

If you are drawing this repository, the most important boxes and arrows are:

- Application code -> public bbolt API
- Public API -> transaction engine
- Transaction engine -> bucket/cursor/node/page/freelist internals
- Internals -> mmap’d database file
- Internals -> OS-specific locking/sync/memory primitives
- CLI -> public API and, for low-level commands, surgeon/internal inspection helpers
- Tests -> all layers, especially transaction correctness, crash behavior, and platform behavior

The key story of the repository is: a minimal embedded Go API backed by a carefully layered storage engine that treats the database file itself as the primary durable state, with strong boundaries around transactions, platform dependencies, and diagnostic tooling.

Main entry points: [`db.go`](https://github.com/adityagaddhyan/bbolt/blob/HEAD/db.go) (DB), [`tx.go`](https://github.com/adityagaddhyan/bbolt/blob/HEAD/tx.go) (Tx), [`internal/common/page.go`](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/common/page.go) (Page), [`cmd/bbolt/command/command_root.go`](https://github.com/adityagaddhyan/bbolt/blob/HEAD/cmd/bbolt/command/command_root.go) (Commands)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Public API"
  db["DB<br/>public package<br/>[db.go]"]
  bucket["Bucket<br/>tree API<br/>[bucket.go]"]
  cursor["Cursor<br/>traversal<br/>[cursor.go]"]
  compact["Compact<br/>rewrite<br/>[compact.go]"]
end

subgraph "Storage Engine"
  tx["Tx<br/>[tx.go]"]
  node["Node<br/>mutable tree<br/>[node.go]"]
  page["Page<br/>on-disk layout<br/>[page.go]"]
  freelist["Freelist<br/>space manager<br/>[freelist.go]"]
  common["Common<br/>binary layout<br/>[types.go]"]
end

subgraph "Platform Layer"
  oslock["OS Bindings<br/>platform layer<br/>[bolt_unix.go]"]
  mlock["Mlock<br/>memory lock<br/>[mlock_unix.go]"]
  syncer["Sync<br/>[boltsync_unix.go]"]
end

subgraph "Tooling"
  cli["bbolt CLI<br/>command line<br/>[main.go]"]
  commands["Commands<br/>[command_root.go]"]
  surgeon["Surgeon<br/>inspection helpers<br/>[surgeon.go]"]
  guts["Guts CLI<br/>internal bridge<br/>[guts_cli.go]"]
end

subgraph "Quality"
  verify["Verify<br/>integrity checks<br/>[verify.go]"]
  robust["Robustness<br/>fault tests<br/>[main_test.go]"]
  failpoint["Failpoints<br/>fault injection"]
  simulation["Simulation<br/>model tests<br/>[simulation_test.go]"]
  txcheck["Tx Check<br/>validation<br/>[tx_check.go]"]
end

db -->|"begins"| tx
db -->|"exposes"| bucket
db -->|"exposes"| cursor
bucket -->|"uses"| cursor
tx -->|"owns"| bucket
tx -->|"materializes"| node
tx -->|"reads/writes"| page
node -->|"serializes to"| page
cursor -->|"walks"| page
freelist -->|"reuses"| page
tx -->|"allocates from"| freelist
page -->|"defined by"| common
node -->|"depends on"| common
common -->|"supports"| verify
db -->|"depends on"| oslock
db -->|"depends on"| mlock
db -->|"depends on"| syncer
cli -->|"dispatches"| commands
commands -->|"uses"| db
commands -->|"inspects"| surgeon
commands -->|"bridges"| guts
compact -->|"uses"| db
txcheck -->|"validates"| verify
failpoint -->|"stresses"| tx
robust -->|"stress tests"| db
simulation -->|"covers"| freelist
simulation -->|"covers"| tx

click db "https://github.com/adityagaddhyan/bbolt/blob/main/db.go"
click tx "https://github.com/adityagaddhyan/bbolt/blob/main/tx.go"
click bucket "https://github.com/adityagaddhyan/bbolt/blob/main/bucket.go"
click cursor "https://github.com/adityagaddhyan/bbolt/blob/main/cursor.go"
click node "https://github.com/adityagaddhyan/bbolt/blob/main/node.go"
click page "https://github.com/adityagaddhyan/bbolt/blob/main/internal/common/page.go"
click freelist "https://github.com/adityagaddhyan/bbolt/blob/main/internal/freelist/freelist.go"
click compact "https://github.com/adityagaddhyan/bbolt/blob/main/compact.go"
click common "https://github.com/adityagaddhyan/bbolt/blob/main/internal/common/types.go"
click verify "https://github.com/adityagaddhyan/bbolt/blob/main/internal/common/verify.go"
click oslock "https://github.com/adityagaddhyan/bbolt/blob/main/bolt_unix.go"
click mlock "https://github.com/adityagaddhyan/bbolt/blob/main/mlock_unix.go"
click syncer "https://github.com/adityagaddhyan/bbolt/blob/main/boltsync_unix.go"
click cli "https://github.com/adityagaddhyan/bbolt/blob/main/cmd/bbolt/main.go"
click commands "https://github.com/adityagaddhyan/bbolt/blob/main/cmd/bbolt/command/command_root.go"
click surgeon "https://github.com/adityagaddhyan/bbolt/blob/main/internal/surgeon/surgeon.go"
click guts "https://github.com/adityagaddhyan/bbolt/blob/main/internal/guts_cli/guts_cli.go"
click robust "https://github.com/adityagaddhyan/bbolt/blob/main/tests/robustness/main_test.go"
click failpoint "https://github.com/adityagaddhyan/bbolt/blob/main/tests/failpoint/db_failpoint_test.go"
click simulation "https://github.com/adityagaddhyan/bbolt/blob/main/simulation_test.go"
click txcheck "https://github.com/adityagaddhyan/bbolt/blob/main/tx_check.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class db,bucket,cursor,compact toneBlue
class tx,node,page,freelist,common toneAmber
class oslock,mlock,syncer toneMint
class cli,commands,surgeon,guts toneRose
class verify,robust,failpoint,simulation,txcheck toneIndigo
```

## Components

### Public API

Repository root package used by applications and the CLI

- [DB](https://github.com/adityagaddhyan/bbolt/blob/HEAD/db.go): public package · Open/close, mmap, locks, and transaction lifecycle · `db.go`
- [Bucket](https://github.com/adityagaddhyan/bbolt/blob/HEAD/bucket.go): tree API · Namespaced key/value access and nested buckets · `bucket.go`
- [Cursor](https://github.com/adityagaddhyan/bbolt/blob/HEAD/cursor.go): traversal · Ordered tree navigation over keys and buckets · `cursor.go`
- [Compact](https://github.com/adityagaddhyan/bbolt/blob/HEAD/compact.go): rewrite · Database rewriting and compaction support · `compact.go`

### Storage Engine

Internal transaction, tree, and space-management subsystems

- [Tx](https://github.com/adityagaddhyan/bbolt/blob/HEAD/tx.go): txn engine · Read/write transactions, commit, rollback, stats · `tx.go`
- [Node](https://github.com/adityagaddhyan/bbolt/blob/HEAD/node.go): mutable tree · In-memory mutable B+tree node representation · `node.go`
- [Page](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/common/page.go): on-disk layout · Serialized database page and meta format · `internal/common/page.go`
- [Freelist](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/freelist/freelist.go): space manager · Tracks reusable pages and file growth · `internal/freelist/freelist.go`
- [Common](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/common/types.go): binary layout · Shared low-level types, inodes, meta, and unsafe helpers · `internal/common/types.go`

### Platform Layer

OS and architecture specific file and memory bindings

- [OS Bindings](https://github.com/adityagaddhyan/bbolt/blob/HEAD/bolt_unix.go): platform layer · File locking, mmap, sync, and memory locking per platform · `bolt_unix.go`
- [Mlock](https://github.com/adityagaddhyan/bbolt/blob/HEAD/mlock_unix.go): memory lock · Platform-specific memory locking support · `mlock_unix.go`
- [Sync](https://github.com/adityagaddhyan/bbolt/blob/HEAD/boltsync_unix.go): fsync helpers · Platform-specific durable write and sync behavior · `boltsync_unix.go`

### Tooling

CLI and deep inspection helpers

- [bbolt CLI](https://github.com/adityagaddhyan/bbolt/blob/HEAD/cmd/bbolt/main.go): command line · Inspection, maintenance, and operational front-end · `cmd/bbolt/main.go`
- [Commands](https://github.com/adityagaddhyan/bbolt/blob/HEAD/cmd/bbolt/command/command_root.go): subcommands · Inspect, dump, stats, check, compact, and surgery actions · `cmd/bbolt/command/command_root.go`
- [Surgeon](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/surgeon/surgeon.go): inspection helpers · Deep file-format inspection and surgical maintenance · `internal/surgeon/surgeon.go`
- [Guts CLI](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/guts_cli/guts_cli.go): internal bridge · Helper plumbing for low-level CLI access · `internal/guts_cli/guts_cli.go`

### Quality

Validation, simulations, and specialized test harnesses

- [Verify](https://github.com/adityagaddhyan/bbolt/blob/HEAD/internal/common/verify.go): integrity checks · Structural verification and corruption detection · `internal/common/verify.go`
- [Robustness](https://github.com/adityagaddhyan/bbolt/blob/HEAD/tests/robustness/main_test.go): fault tests · Crash consistency, flaky I/O, and power-failure scenarios · `tests/robustness/main_test.go`
- [Failpoints](https://github.com/adityagaddhyan/bbolt/blob/HEAD/tests/failpoint/db_failpoint_test.go): fault injection · Injected failure paths for transactional behavior · `tests/failpoint/db_failpoint_test.go`
- [Simulation](https://github.com/adityagaddhyan/bbolt/blob/HEAD/simulation_test.go): model tests · Behavioral and allocation simulations for engine invariants · `simulation_test.go`
- [Tx Check](https://github.com/adityagaddhyan/bbolt/blob/HEAD/tx_check.go): validation · Transaction-level invariant checking and diagnostics · `tx_check.go`

## Connections

- DB → Tx: begins · Database opens and manages transaction lifecycle
- DB → Bucket: exposes · Public API for namespaced key/value access
- DB → Cursor: exposes · Ordered traversal interface for callers
- Bucket → Cursor: uses · Bucket operations navigate through cursors
- Tx → Bucket: owns · Transactions present bucket views over a snapshot
- Tx → Node: materializes · Writable transactions build mutable nodes from pages
- Tx → Page: reads/writes · Commit serializes dirty pages and reads meta/pages
- Node → Page: serializes to · Mutable nodes are flushed into physical pages
- Cursor → Page: walks · Traversal resolves keys through leaf and branch pages
- Freelist → Page: reuses · Tracks freed pages for later allocation
- Tx → Freelist: allocates from · Writes consume free pages before growing the file
- Page → Common: defined by · On-disk layout is centralized in shared low-level types
- Node → Common: depends on · Mutable tree structures share binary record definitions
- Common → Verify: supports · Common layout definitions feed integrity checks
- DB → OS Bindings: depends on · Database opening and lifecycle depend on OS primitives
- DB → Mlock: depends on · Optional memory locking is platform specific
- DB → Sync: depends on · Durability relies on platform sync behavior
- bbolt CLI → Commands: dispatches · Main binary routes into subcommands
- Commands → DB: uses · Normal commands operate through the public library API
- Commands → Surgeon: inspects · Low-level commands use deep inspection helpers
- Commands → Guts CLI: bridges · CLI plumbing reaches internal file-format helpers
- Compact → DB: uses · Compaction is built on the database read/write API
- Tx Check → Verify: validates · Transaction checks rely on shared verification routines
- Failpoints → Tx: stresses · Injected failures target transaction commit and rollback
- Robustness → DB: stress tests · Crash-consistency scenarios exercise database behavior
- Simulation → Freelist: covers · Simulations exercise allocation and reclamation behavior
- Simulation → Tx: covers · Simulations exercise transaction invariants
