How surrealdb/surrealkv works
SurrealKV is an embedded LSM-tree key-value engine. The principal workflow accepts Tree operations, commits batches through an asynchronous pipeline, records them in the WAL and memtable, and later flushes and compacts sorted tables; reads and iteration consult the engine’s storage structures. Supporting branches include recovery, snapshots/checkpoints, encryption, browser storage, and legacy-format migration. The source index and excerpts support the relationships shown; several implementation details are unsampled, so uncertain links are omitted.
Main entry points: lsm.rs LSM coordinator · lib.rs Tree API · pipeline.rs Commit pipeline · mod.rs Memtable
Components 18
API and transactions
- Tree API
src/lib.rs - Transactions
src/transaction.rs
LSM engine
- Commit pipeline
src/ring/pipeline.rs - Write batches
src/batch.rs - LSM coordinator
src/lsm.rs - Memtable
src/memtable/mod.rs - Compaction
src/compaction/compactor.rs - Levels manifest
src/levels/mod.rs
Persistence and files
- Write-ahead log
src/wal/manager.rs - WAL recovery
src/wal/recovery.rs - Sorted tables
src/sstable/table.rs - Value log
src/vlog.rs - Virtual filesystem
src/vfs/mod.rs
Storage capabilities
- Snapshots and checkpoints
src/checkpoint.rs - At-rest encryption
src/tde/mod.rs - Format migration
src/migrate/mod.rs - Browser OPFS storage
src/storage/opfs.rs
Other components
- Application
Connections 25
- Application → Tree API: uses
- Tree API → Transactions: offers
src/lib.rs - Tree API → LSM coordinator: opens
src/lib.rs - Transactions → Commit pipeline: submits commits
- Commit pipeline → Write batches: encodes
src/ring/pipeline.rs - Commit pipeline → LSM coordinator: commits
src/ring/pipeline.rs - Commit pipeline → Memtable: updates
src/ring/pipeline.rs - Commit pipeline → Write-ahead log: appends
- LSM coordinator → Memtable: manages
src/lsm.rs - LSM coordinator → Write-ahead log: recovers from
src/lsm.rs - LSM coordinator → Compaction: schedules
src/lsm.rs - Compaction → Sorted tables: writes tables
src/compaction/compactor.rs - Compaction → Levels manifest: updates
src/compaction/compactor.rs - LSM coordinator → Levels manifest: uses levels
src/lsm.rs - WAL recovery → Write-ahead log: reads records
src/wal/recovery.rs - WAL recovery → Memtable: replays batches
src/wal/recovery.rs - Memtable → Sorted tables: flushes to
src/memtable/mod.rs - Sorted tables → Virtual filesystem: reads and writes
src/sstable/table.rs - Write-ahead log → Virtual filesystem: uses filesystem
src/wal/manager.rs - Value log → Virtual filesystem: uses filesystem
src/vlog.rs - Tree API → Snapshots and checkpoints: exposes
src/lib.rs - Snapshots and checkpoints → LSM coordinator: coordinates
src/checkpoint.rs - Tree API → At-rest encryption: configures
src/lib.rs - LSM coordinator → Format migration: supports migration
src/lsm.rs - Virtual filesystem → Browser OPFS storage: selects on browser
src/vfs/mod.rs