How cockroachdb/pebble works
1. What this repository is
- Pebble is an embeddable Go key-value store modeled after LevelDB/RocksDB.
- The top-level pebble package exposes the database API; most implementation details are split into storage-engine subsystems under sstable, wal, vfs, objstorage, internal/*, and tooling packages.
2. Primary runtime boundary: the DB API
- The root package contains the user-facing surface: Open, DB, Options, Batch, Iterator, Snapshot, Get/Set/Delete, ingest, compaction, checkpoint, and recovery logic.
- This layer coordinates all subsystems and is the main boundary between application code and the storage engine internals.
3. Write path
- Writes enter through batches or direct mutations, are sequenced in memory, and are first durably recorded in the WAL.
- They are then applied to the in-memory memtable structure, with flushable and indexed-batch variants supported.
- Once memory pressure or flush triggers occur, mutable state is turned into SSTables and the manifest/version state is updated.
4. Read path
- Reads consult the current read state: memtables, immutable memtables, L0 sublevels, and on-disk SSTables.
- Iteration is a first-class feature; there are multiple iterator layers for point lookups, merged iteration across levels, prefix/range-key handling, and external iteration over on-disk structures.
- Snapshots and sequence numbers control visibility, so reads are transactionally consistent without full transactions.
5. On-disk data model: SSTables
- The sstable package is a major subsystem and contains the file format, block encoding, readers, writers, filters, properties, and specialized layouts.
- Subpackages split the table format into physical block types (block, rowblk, colblk, valblk, blob) and filter/index structures.
- This package is responsible for encoding persistent sorted runs and for reading them efficiently during queries and compactions.
6. Metadata and versioning
- manifest, version_set.go, version.go, and related code maintain the logical view of the LSM: which files exist, which levels they belong to, and what key ranges they cover.
- The manifest is copy-on-write style metadata, used to reconstruct state after restart and to drive compaction planning.
- Format major version logic gates file-format transitions and migrations.
7. Compaction subsystem
- Compaction is a central pipeline: picking, scheduling, execution, tombstone elision, delete-only compactions, value separation handling, and level rewriting.
- The root compaction code coordinates with internal/compact, compaction_*, read_compaction_queue.go, and level_checker.go.
- This is the main mechanism that maintains LSM invariants, bounds read amplification, and rewrites formats over time.
8. Write durability and recovery
- The wal package manages log writing, reading, recycling, and failover behavior.
- recovery.go, checkpoint.go, and open.go use WAL + manifest state to reconstruct the DB after crash or startup.
- The repo also includes replay/capture tools for reproducing workloads and diagnosing recovery/IO behavior.
9. Storage abstraction layers
- vfs abstracts filesystem access, including OS-specific implementations, in-memory filesystems, atomic marker files, disk-health/full-disk handling, and testing fakes.
- objstorage abstracts object storage and file access patterns, including local, remote, shared-cache, and readahead support.
- These layers are important boundaries because Pebble can operate against different file backends and testing environments without changing the core engine.
10. Caching and memory management
- internal/cache, file_cache.go, and related packages manage block cache behavior and reference counting.
- The engine also has internal allocators and buffer management for logs, blocks, and table reads.
- Cache and readahead sit on the read path between SSTable readers and the underlying storage.
11. Advanced keyspace features
- Pebble supports range deletions, range keys, merge operators, single-delete semantics, blob/value separation, and table-level bloom filters.
- These features flow through compaction, iteration, and SSTable encoding, so they are not isolated modules but cross-cutting concerns.
- valsep, rangedel, rangekey, and sstable/blob are the main specialized feature packages.
12. Internal building blocks
- The internal/* tree contains reusable engine primitives: comparers, iterators, key-span logic, manifests, metrics, compression, invalidation, lock/close helpers, and test harnesses.
- These packages are mostly not public API; they support the top-level engine while keeping the main package smaller.
- Many have targeted testdata and data-driven tests because correctness depends on subtle file-format and iterator behavior.
13. Tooling and observability
- cmd/pebble is a CLI for inspecting, replaying, benchmarking, scanning, and testing databases and SSTables.
- tool/ provides analysis and inspection utilities for manifests, WALs, LSM layout, blob files, and database properties.
- docs/, metamorphic/, replay/, and benchmark packages support performance analysis, regression testing, and architecture validation rather than production runtime.
14. How the pieces fit together
- At a high level, the system is an LSM tree: mutable writes land in memory and WAL, flushed data becomes SSTables, version/manifest metadata tracks file sets, and compactions continuously rewrite files to preserve read performance.
- Reads merge multiple sources in priority order, with snapshots and sequence numbers providing visibility rules.
- The most important diagram boundaries are: user API → DB core → WAL/memtable/manifest/compaction → SSTable readers/writers → filesystem/object storage/cache.
Main entry points: db.go DB · table.go SSTables · mem_table.go Write Path · compaction.go Compaction
Components 21
DB API
Public database surface and lifecycle control.
- DB · User-facing Pebble database surface: open, read, write, batch, snapshot, ingest, checkpoint, and recovery entrypoints.
db.go - Options · Configuration and tuning for DB behavior and subsystem wiring.
options.go - Batch · Write batching and mutation encoding for user operations.
batch.go - Iterators · Point lookup and iteration surface, including snapshots and merged iteration.
iterator.go
Engine Core
Write path, read path, compaction, and metadata coordination.
- Write Path · Coordinates sequencing, WAL append, memtable application, and flush triggers.
mem_table.go - Memtable · Mutable in-memory sorted structure plus immutable/flushable variants.
mem_table.go - Manifest · Logical LSM state: file sets, levels, key ranges, and restart reconstruction.
version_set.go - Compaction · Picking, scheduling, execution, tombstone elision, and level rewriting.
compaction.go - Read State · Current visibility state used by reads: memtables, sublevels, and SSTables.
read_state.go - Internal Libs · Reusable internal building blocks for comparers, key spans, manifests, metrics, compression, and iterator helpers.
internal/base/options.go
Storage Backends
Durability, table files, filesystem, object storage, and caching.
- WAL · Write-ahead log management for persistence, replay, recycling, and failover.
wal/wal.go - SSTables · Sorted-run file format, block layouts, readers, writers, filters, and properties.
sstable/table.go - Block Layouts · Physical SSTable layouts such as row blocks, column blocks, value blocks, and blob handling.
sstable/rowblk/rowblk_writer.go - Filters · Bloom and binary-fuse table filters used to accelerate negative lookups.
sstable/tablefilters/table_filters.go - VFS · Filesystem abstraction with OS-specific backends, in-memory filesystems, locking, and atomic markers.
vfs/vfs.go - Obj Storage · Storage provider abstraction for local, remote, shared-cache, and readahead access patterns.
objstorage/objstorage.go - Cache · Block cache and reference counting between SSTable readers and underlying storage.
file_cache.go
Tooling
Inspection, replay, benchmarks, and metamorphic validation.
- CLI · Command-line interface for inspection, replay, scanning, benchmarking, and test utilities.
cmd/pebble/main.go - Tooling · Database and manifest inspection utilities, SSTable/WAL viewers, and layout analysis tools.
tool/tool.go - Replay · Workload capture and replay tooling for reproducing performance and recovery behavior.
replay/replay.go - Metamorphic · Property-style test harness and workload generator for cross-version and behavioral validation.
metamorphic/meta.go
Connections 26
- DB → Options: configures · DB startup and runtime behavior are governed by options.
- DB → Batch: accepts · User mutations arrive through batches and direct write operations.
- DB → Iterators: exposes · DB provides read, get, snapshot, and iterator entrypoints.
- DB → Write Path: routes writes · Public mutations are coordinated through the write path.
- DB → Read State: serves reads · Lookups and iterators consult the current read state.
- Write Path → WAL: durable first · Writes are appended to the WAL before in-memory application.
- Write Path → Memtable: applies to · Sequenced mutations are applied into the mutable memtable.
- Write Path → Manifest: updates metadata · Flush and ingest operations advance version and file-set state.
- Memtable → Read State: feeds · Reads consult mutable and immutable in-memory tables.
- Read State → SSTables: reads from · Read state includes on-disk SSTables across levels and sublevels.
- Compaction → Manifest: rewrites · Compaction produces new file layouts and updates version metadata.
- Compaction → SSTables: produces · Compaction reads and writes SSTables during LSM rewriting.
- SSTables → Block Layouts: uses · Table files are organized into specialized block encodings.
- SSTables → Filters: indexes with · Table readers and writers use filters for fast negative lookups.
- SSTables → Obj Storage: stored via · Table access can be backed by object storage providers.
- Obj Storage → VFS: backs onto · Local object access and file operations are implemented through VFS backends.
- Cache → SSTables: accelerates · The block cache sits on the SSTable read path.
- Cache → Obj Storage: prefetches · Cache and readahead mediate object reads from storage.
- Internal Libs → DB: supports · Core engine primitives underpin the public API and runtime coordination.
- Internal Libs → SSTables: supports · Shared internals also support table format, iterators, and encoding logic.
- CLI → DB: drives · CLI commands exercise the Pebble database and its state transitions.
- Tooling → Manifest: inspects · Tooling reads manifests and version state for analysis.
- Tooling → SSTables: inspects · Tooling can inspect SSTables, layouts, and properties.
- Replay → WAL: replays · Workload capture and replay depend on WAL and recovery behavior.
- Metamorphic → DB: validates · Metamorphic tests exercise database behavior under varied operation sequences.
- Metamorphic → Compaction: stresses · The harness probes compaction, iteration, and versioning edge cases.