# cockroachdb/pebble architecture

> Architecture diagram of the GitHub repository cockroachdb/pebble, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cockroachdb/pebble
- Repository: https://github.com/cockroachdb/pebble

## Overview

1. **What this repository is**
   - Pebble is an embeddable Go key-value store modeled after LevelDB/RocksDB.
   - The top-level `pebble` package exposes the database API; most implementation details are split into storage-engine subsystems under `sstable`, `wal`, `vfs`, `objstorage`, `internal/*`, and tooling packages.

2. **Primary runtime boundary: the DB API**
   - The root package contains the user-facing surface: `Open`, `DB`, `Options`, `Batch`, `Iterator`, `Snapshot`, `Get/Set/Delete`, ingest, compaction, checkpoint, and recovery logic.
   - This layer coordinates all subsystems and is the main boundary between application code and the storage engine internals.

3. **Write path**
   - Writes enter through batches or direct mutations, are sequenced in memory, and are first durably recorded in the **WAL**.
   - They are then applied to the in-memory **memtable** structure, with flushable and indexed-batch variants supported.
   - Once memory pressure or flush triggers occur, mutable state is turned into SSTables and the manifest/version state is updated.

4. **Read path**
   - Reads consult the current read state: memtables, immutable memtables, L0 sublevels, and on-disk SSTables.
   - Iteration is a first-class feature; there are multiple iterator layers for point lookups, merged iteration across levels, prefix/range-key handling, and external iteration over on-disk structures.
   - Snapshots and sequence numbers control visibility, so reads are transactionally consistent without full transactions.

5. **On-disk data model: SSTables**
   - The `sstable` package is a major subsystem and contains the file format, block encoding, readers, writers, filters, properties, and specialized layouts.
   - Subpackages split the table format into physical block types (`block`, `rowblk`, `colblk`, `valblk`, `blob`) and filter/index structures.
   - This package is responsible for encoding persistent sorted runs and for reading them efficiently during queries and compactions.

6. **Metadata and versioning**
   - `manifest`, `version_set.go`, `version.go`, and related code maintain the logical view of the LSM: which files exist, which levels they belong to, and what key ranges they cover.
   - The manifest is copy-on-write style metadata, used to reconstruct state after restart and to drive compaction planning.
   - Format major version logic gates file-format transitions and migrations.

7. **Compaction subsystem**
   - Compaction is a central pipeline: picking, scheduling, execution, tombstone elision, delete-only compactions, value separation handling, and level rewriting.
   - The root compaction code coordinates with `internal/compact`, `compaction_*`, `read_compaction_queue.go`, and `level_checker.go`.
   - This is the main mechanism that maintains LSM invariants, bounds read amplification, and rewrites formats over time.

8. **Write durability and recovery**
   - The `wal` package manages log writing, reading, recycling, and failover behavior.
   - `recovery.go`, `checkpoint.go`, and `open.go` use WAL + manifest state to reconstruct the DB after crash or startup.
   - The repo also includes replay/capture tools for reproducing workloads and diagnosing recovery/IO behavior.

9. **Storage abstraction layers**
   - `vfs` abstracts filesystem access, including OS-specific implementations, in-memory filesystems, atomic marker files, disk-health/full-disk handling, and testing fakes.
   - `objstorage` abstracts object storage and file access patterns, including local, remote, shared-cache, and readahead support.
   - These layers are important boundaries because Pebble can operate against different file backends and testing environments without changing the core engine.

10. **Caching and memory management**
   - `internal/cache`, `file_cache.go`, and related packages manage block cache behavior and reference counting.
   - The engine also has internal allocators and buffer management for logs, blocks, and table reads.
   - Cache and readahead sit on the read path between SSTable readers and the underlying storage.

11. **Advanced keyspace features**
   - Pebble supports range deletions, range keys, merge operators, single-delete semantics, blob/value separation, and table-level bloom filters.
   - These features flow through compaction, iteration, and SSTable encoding, so they are not isolated modules but cross-cutting concerns.
   - `valsep`, `rangedel`, `rangekey`, and `sstable/blob` are the main specialized feature packages.

12. **Internal building blocks**
   - The `internal/*` tree contains reusable engine primitives: comparers, iterators, key-span logic, manifests, metrics, compression, invalidation, lock/close helpers, and test harnesses.
   - These packages are mostly not public API; they support the top-level engine while keeping the main package smaller.
   - Many have targeted testdata and data-driven tests because correctness depends on subtle file-format and iterator behavior.

13. **Tooling and observability**
   - `cmd/pebble` is a CLI for inspecting, replaying, benchmarking, scanning, and testing databases and SSTables.
   - `tool/` provides analysis and inspection utilities for manifests, WALs, LSM layout, blob files, and database properties.
   - `docs/`, `metamorphic/`, `replay/`, and benchmark packages support performance analysis, regression testing, and architecture validation rather than production runtime.

14. **How the pieces fit together**
   - At a high level, the system is an LSM tree: mutable writes land in memory and WAL, flushed data becomes SSTables, version/manifest metadata tracks file sets, and compactions continuously rewrite files to preserve read performance.
   - Reads merge multiple sources in priority order, with snapshots and sequence numbers providing visibility rules.
   - The most important diagram boundaries are: user API → DB core → WAL/memtable/manifest/compaction → SSTable readers/writers → filesystem/object storage/cache.

Main entry points: [`db.go`](https://github.com/cockroachdb/pebble/blob/master/db.go) (DB), [`sstable/table.go`](https://github.com/cockroachdb/pebble/blob/master/sstable/table.go) (SSTables), [`mem_table.go`](https://github.com/cockroachdb/pebble/blob/master/mem_table.go) (Write Path), [`compaction.go`](https://github.com/cockroachdb/pebble/blob/master/compaction.go) (Compaction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["DB API"]
  node_db_api["DB<br/>public API<br/>[db.go]"]
  node_options["Options<br/>config<br/>[options.go]"]
  node_batch["Batch<br/>mutations<br/>[batch.go]"]
  node_read_api["Iterators<br/>read API<br/>[iterator.go]"]
end

subgraph group_core["Engine Core"]
  node_write_coord["Write Path<br/>coordination<br/>[mem_table.go]"]
  node_memtable{{"Memtable<br/>in-memory index<br/>[mem_table.go]"}}
  node_manifest[("Manifest<br/>version metadata<br/>[version_set.go]")]
  node_compaction["Compaction<br/>rewrite pipeline<br/>[compaction.go]"]
  node_read_state["Read State<br/>lsm view<br/>[read_state.go]"]
  node_internal_libs["Internal Libs<br/>engine primitives<br/>[options.go]"]
end

subgraph group_storage["Storage Backends"]
  node_wal_pkg["WAL<br/>durability log<br/>[wal.go]"]
  node_sstable["SSTables<br/>table format<br/>[table.go]"]
  node_block_formats["Block Layouts<br/>sstable internals<br/>[rowblk_writer.go]"]
  node_table_filters{{"Filters<br/>[table_filters.go]"}}
  node_vfs["VFS<br/>filesystem abstraction<br/>[vfs.go]"]
  node_objstorage["Obj Storage<br/>object storage<br/>[objstorage.go]"]
  node_cache{{"Cache<br/>[file_cache.go]"}}
end

subgraph group_tools["Tooling"]
  node_cmd_pebble["CLI<br/>inspection tool<br/>[main.go]"]
  node_tooling["Tooling<br/>analysis tools<br/>[tool.go]"]
  node_replay["Replay<br/>workload capture<br/>[replay.go]"]
  node_metamorphic["Metamorphic<br/>validation harness<br/>[meta.go]"]
end

node_db_api -->|"configures"| node_options
node_db_api -->|"accepts"| node_batch
node_db_api -->|"exposes"| node_read_api
node_db_api -->|"routes writes"| node_write_coord
node_db_api -->|"serves reads"| node_read_state
node_write_coord -->|"durable first"| node_wal_pkg
node_write_coord -->|"applies to"| node_memtable
node_write_coord -->|"updates metadata"| node_manifest
node_memtable -->|"feeds"| node_read_state
node_read_state -->|"reads from"| node_sstable
node_compaction -->|"rewrites"| node_manifest
node_compaction -->|"produces"| node_sstable
node_sstable -->|"uses"| node_block_formats
node_sstable -->|"indexes with"| node_table_filters
node_sstable -->|"stored via"| node_objstorage
node_objstorage -->|"backs onto"| node_vfs
node_cache -->|"accelerates"| node_sstable
node_cache -->|"prefetches"| node_objstorage
node_internal_libs -->|"supports"| node_db_api
node_internal_libs -->|"supports"| node_sstable
node_cmd_pebble -->|"drives"| node_db_api
node_tooling -->|"inspects"| node_manifest
node_tooling -->|"inspects"| node_sstable
node_replay -->|"replays"| node_wal_pkg
node_metamorphic -->|"validates"| node_db_api
node_metamorphic -->|"stresses"| node_compaction

click node_db_api "https://github.com/cockroachdb/pebble/blob/master/db.go"
click node_options "https://github.com/cockroachdb/pebble/blob/master/options.go"
click node_batch "https://github.com/cockroachdb/pebble/blob/master/batch.go"
click node_read_api "https://github.com/cockroachdb/pebble/blob/master/iterator.go"
click node_write_coord "https://github.com/cockroachdb/pebble/blob/master/mem_table.go"
click node_memtable "https://github.com/cockroachdb/pebble/blob/master/mem_table.go"
click node_wal_pkg "https://github.com/cockroachdb/pebble/blob/master/wal/wal.go"
click node_manifest "https://github.com/cockroachdb/pebble/blob/master/version_set.go"
click node_compaction "https://github.com/cockroachdb/pebble/blob/master/compaction.go"
click node_read_state "https://github.com/cockroachdb/pebble/blob/master/read_state.go"
click node_sstable "https://github.com/cockroachdb/pebble/blob/master/sstable/table.go"
click node_block_formats "https://github.com/cockroachdb/pebble/blob/master/sstable/rowblk/rowblk_writer.go"
click node_table_filters "https://github.com/cockroachdb/pebble/blob/master/sstable/tablefilters/table_filters.go"
click node_vfs "https://github.com/cockroachdb/pebble/blob/master/vfs/vfs.go"
click node_objstorage "https://github.com/cockroachdb/pebble/blob/master/objstorage/objstorage.go"
click node_cache "https://github.com/cockroachdb/pebble/blob/master/file_cache.go"
click node_internal_libs "https://github.com/cockroachdb/pebble/blob/master/internal/base/options.go"
click node_cmd_pebble "https://github.com/cockroachdb/pebble/blob/master/cmd/pebble/main.go"
click node_tooling "https://github.com/cockroachdb/pebble/blob/master/tool/tool.go"
click node_replay "https://github.com/cockroachdb/pebble/blob/master/replay/replay.go"
click node_metamorphic "https://github.com/cockroachdb/pebble/blob/master/metamorphic/meta.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_db_api,node_options,node_batch,node_read_api toneBlue
class node_write_coord,node_memtable,node_manifest,node_compaction,node_read_state,node_internal_libs toneAmber
class node_wal_pkg,node_sstable,node_block_formats,node_table_filters,node_vfs,node_objstorage,node_cache toneMint
class node_cmd_pebble,node_tooling,node_replay,node_metamorphic toneRose
```

## Components

### DB API

Public database surface and lifecycle control.

- [DB](https://github.com/cockroachdb/pebble/blob/master/db.go): public API · User-facing Pebble database surface: open, read, write, batch, snapshot, ingest, checkpoint, and recovery entrypoints. · `db.go`
- [Options](https://github.com/cockroachdb/pebble/blob/master/options.go): config · Configuration and tuning for DB behavior and subsystem wiring. · `options.go`
- [Batch](https://github.com/cockroachdb/pebble/blob/master/batch.go): mutations · Write batching and mutation encoding for user operations. · `batch.go`
- [Iterators](https://github.com/cockroachdb/pebble/blob/master/iterator.go): read API · Point lookup and iteration surface, including snapshots and merged iteration. · `iterator.go`

### Engine Core

Write path, read path, compaction, and metadata coordination.

- [Write Path](https://github.com/cockroachdb/pebble/blob/master/mem_table.go): coordination · Coordinates sequencing, WAL append, memtable application, and flush triggers. · `mem_table.go`
- [Memtable](https://github.com/cockroachdb/pebble/blob/master/mem_table.go): in-memory index · Mutable in-memory sorted structure plus immutable/flushable variants. · `mem_table.go`
- [Manifest](https://github.com/cockroachdb/pebble/blob/master/version_set.go): version metadata · Logical LSM state: file sets, levels, key ranges, and restart reconstruction. · `version_set.go`
- [Compaction](https://github.com/cockroachdb/pebble/blob/master/compaction.go): rewrite pipeline · Picking, scheduling, execution, tombstone elision, and level rewriting. · `compaction.go`
- [Read State](https://github.com/cockroachdb/pebble/blob/master/read_state.go): lsm view · Current visibility state used by reads: memtables, sublevels, and SSTables. · `read_state.go`
- [Internal Libs](https://github.com/cockroachdb/pebble/blob/master/internal/base/options.go): engine primitives · Reusable internal building blocks for comparers, key spans, manifests, metrics, compression, and iterator helpers. · `internal/base/options.go`

### Storage Backends

Durability, table files, filesystem, object storage, and caching.

- [WAL](https://github.com/cockroachdb/pebble/blob/master/wal/wal.go): durability log · Write-ahead log management for persistence, replay, recycling, and failover. · `wal/wal.go`
- [SSTables](https://github.com/cockroachdb/pebble/blob/master/sstable/table.go): table format · Sorted-run file format, block layouts, readers, writers, filters, and properties. · `sstable/table.go`
- [Block Layouts](https://github.com/cockroachdb/pebble/blob/master/sstable/rowblk/rowblk_writer.go): sstable internals · Physical SSTable layouts such as row blocks, column blocks, value blocks, and blob handling. · `sstable/rowblk/rowblk_writer.go`
- [Filters](https://github.com/cockroachdb/pebble/blob/master/sstable/tablefilters/table_filters.go): table filters · Bloom and binary-fuse table filters used to accelerate negative lookups. · `sstable/tablefilters/table_filters.go`
- [VFS](https://github.com/cockroachdb/pebble/blob/master/vfs/vfs.go): filesystem abstraction · Filesystem abstraction with OS-specific backends, in-memory filesystems, locking, and atomic markers. · `vfs/vfs.go`
- [Obj Storage](https://github.com/cockroachdb/pebble/blob/master/objstorage/objstorage.go): object storage · Storage provider abstraction for local, remote, shared-cache, and readahead access patterns. · `objstorage/objstorage.go`
- [Cache](https://github.com/cockroachdb/pebble/blob/master/file_cache.go): block cache · Block cache and reference counting between SSTable readers and underlying storage. · `file_cache.go`

### Tooling

Inspection, replay, benchmarks, and metamorphic validation.

- [CLI](https://github.com/cockroachdb/pebble/blob/master/cmd/pebble/main.go): inspection tool · Command-line interface for inspection, replay, scanning, benchmarking, and test utilities. · `cmd/pebble/main.go`
- [Tooling](https://github.com/cockroachdb/pebble/blob/master/tool/tool.go): analysis tools · Database and manifest inspection utilities, SSTable/WAL viewers, and layout analysis tools. · `tool/tool.go`
- [Replay](https://github.com/cockroachdb/pebble/blob/master/replay/replay.go): workload capture · Workload capture and replay tooling for reproducing performance and recovery behavior. · `replay/replay.go`
- [Metamorphic](https://github.com/cockroachdb/pebble/blob/master/metamorphic/meta.go): validation harness · Property-style test harness and workload generator for cross-version and behavioral validation. · `metamorphic/meta.go`

## Connections

- DB → Options: configures · DB startup and runtime behavior are governed by options.
- DB → Batch: accepts · User mutations arrive through batches and direct write operations.
- DB → Iterators: exposes · DB provides read, get, snapshot, and iterator entrypoints.
- DB → Write Path: routes writes · Public mutations are coordinated through the write path.
- DB → Read State: serves reads · Lookups and iterators consult the current read state.
- Write Path → WAL: durable first · Writes are appended to the WAL before in-memory application.
- Write Path → Memtable: applies to · Sequenced mutations are applied into the mutable memtable.
- Write Path → Manifest: updates metadata · Flush and ingest operations advance version and file-set state.
- Memtable → Read State: feeds · Reads consult mutable and immutable in-memory tables.
- Read State → SSTables: reads from · Read state includes on-disk SSTables across levels and sublevels.
- Compaction → Manifest: rewrites · Compaction produces new file layouts and updates version metadata.
- Compaction → SSTables: produces · Compaction reads and writes SSTables during LSM rewriting.
- SSTables → Block Layouts: uses · Table files are organized into specialized block encodings.
- SSTables → Filters: indexes with · Table readers and writers use filters for fast negative lookups.
- SSTables → Obj Storage: stored via · Table access can be backed by object storage providers.
- Obj Storage → VFS: backs onto · Local object access and file operations are implemented through VFS backends.
- Cache → SSTables: accelerates · The block cache sits on the SSTable read path.
- Cache → Obj Storage: prefetches · Cache and readahead mediate object reads from storage.
- Internal Libs → DB: supports · Core engine primitives underpin the public API and runtime coordination.
- Internal Libs → SSTables: supports · Shared internals also support table format, iterators, and encoding logic.
- CLI → DB: drives · CLI commands exercise the Pebble database and its state transitions.
- Tooling → Manifest: inspects · Tooling reads manifests and version state for analysis.
- Tooling → SSTables: inspects · Tooling can inspect SSTables, layouts, and properties.
- Replay → WAL: replays · Workload capture and replay depend on WAL and recovery behavior.
- Metamorphic → DB: validates · Metamorphic tests exercise database behavior under varied operation sequences.
- Metamorphic → Compaction: stresses · The harness probes compaction, iteration, and versioning edge cases.
