# surrealdb/surrealkv architecture

> Architecture diagram of the GitHub repository surrealdb/surrealkv, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/surrealdb/surrealkv
- Repository: https://github.com/surrealdb/surrealkv

## Overview

SurrealKV is an embedded LSM-tree key-value engine. The principal workflow accepts Tree operations, commits batches through an asynchronous pipeline, records them in the WAL and memtable, and later flushes and compacts sorted tables; reads and iteration consult the engine’s storage structures. Supporting branches include recovery, snapshots/checkpoints, encryption, browser storage, and legacy-format migration. The source index and excerpts support the relationships shown; several implementation details are unsampled, so uncertain links are omitted.

Main entry points: [`src/lsm.rs`](https://github.com/surrealdb/surrealkv/blob/main/src/lsm.rs) (LSM coordinator), [`src/lib.rs`](https://github.com/surrealdb/surrealkv/blob/main/src/lib.rs) (Tree API), [`src/ring/pipeline.rs`](https://github.com/surrealdb/surrealkv/blob/main/src/ring/pipeline.rs) (Commit pipeline), [`src/memtable/mod.rs`](https://github.com/surrealdb/surrealkv/blob/main/src/memtable/mod.rs) (Memtable)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and transactions"]
  node_tree["Tree API<br/>[lib.rs]"]
  node_transactions["Transactions<br/>[transaction.rs]"]
end

subgraph group_engine["LSM engine"]
  node_commit["Commit pipeline<br/>[pipeline.rs]"]
  node_batch["Write batches<br/>[batch.rs]"]
  node_lsm["LSM coordinator<br/>[lsm.rs]"]
  node_memtable["Memtable<br/>[mod.rs]"]
  node_compaction["Compaction<br/>[compactor.rs]"]
  node_manifest["Levels manifest<br/>[mod.rs]"]
end

subgraph group_persistence["Persistence and files"]
  node_wal["Write-ahead log<br/>[manager.rs]"]
  node_recovery["WAL recovery<br/>[recovery.rs]"]
  node_sst["Sorted tables<br/>[table.rs]"]
  node_vlog["Value log<br/>[vlog.rs]"]
  node_files["Virtual filesystem<br/>[mod.rs]"]
end

subgraph group_capabilities["Storage capabilities"]
  node_checkpoint["Snapshots and checkpoints<br/>[checkpoint.rs]"]
  node_crypto["At-rest encryption<br/>[mod.rs]"]
  node_migration["Format migration<br/>[mod.rs]"]
  node_opfs["Browser OPFS storage<br/>[opfs.rs]"]
end

node_user(("Application"))

node_user -->|"uses"| node_tree
node_tree -->|"offers"| node_transactions
node_tree -->|"opens"| node_lsm
node_transactions -->|"submits commits"| node_commit
node_commit -->|"encodes"| node_batch
node_commit -->|"commits"| node_lsm
node_commit -->|"updates"| node_memtable
node_commit -.->|"appends"| node_wal
node_lsm -->|"manages"| node_memtable
node_lsm -->|"recovers from"| node_wal
node_lsm -->|"schedules"| node_compaction
node_compaction -->|"writes tables"| node_sst
node_compaction -->|"updates"| node_manifest
node_lsm -->|"uses levels"| node_manifest
node_recovery -->|"reads records"| node_wal
node_recovery -->|"replays batches"| node_memtable
node_memtable -->|"flushes to"| node_sst
node_sst -->|"reads and writes"| node_files
node_wal -->|"uses filesystem"| node_files
node_vlog -->|"uses filesystem"| node_files
node_tree -->|"exposes"| node_checkpoint
node_checkpoint -->|"coordinates"| node_lsm
node_tree -->|"configures"| node_crypto
node_lsm -->|"supports migration"| node_migration
node_files -.->|"selects on browser"| node_opfs

click node_tree "https://github.com/surrealdb/surrealkv/blob/main/src/lib.rs"
click node_transactions "https://github.com/surrealdb/surrealkv/blob/main/src/transaction.rs"
click node_commit "https://github.com/surrealdb/surrealkv/blob/main/src/ring/pipeline.rs"
click node_batch "https://github.com/surrealdb/surrealkv/blob/main/src/batch.rs"
click node_lsm "https://github.com/surrealdb/surrealkv/blob/main/src/lsm.rs"
click node_memtable "https://github.com/surrealdb/surrealkv/blob/main/src/memtable/mod.rs"
click node_compaction "https://github.com/surrealdb/surrealkv/blob/main/src/compaction/compactor.rs"
click node_manifest "https://github.com/surrealdb/surrealkv/blob/main/src/levels/mod.rs"
click node_wal "https://github.com/surrealdb/surrealkv/blob/main/src/wal/manager.rs"
click node_recovery "https://github.com/surrealdb/surrealkv/blob/main/src/wal/recovery.rs"
click node_sst "https://github.com/surrealdb/surrealkv/blob/main/src/sstable/table.rs"
click node_vlog "https://github.com/surrealdb/surrealkv/blob/main/src/vlog.rs"
click node_files "https://github.com/surrealdb/surrealkv/blob/main/src/vfs/mod.rs"
click node_checkpoint "https://github.com/surrealdb/surrealkv/blob/main/src/checkpoint.rs"
click node_crypto "https://github.com/surrealdb/surrealkv/blob/main/src/tde/mod.rs"
click node_migration "https://github.com/surrealdb/surrealkv/blob/main/src/migrate/mod.rs"
click node_opfs "https://github.com/surrealdb/surrealkv/blob/main/src/storage/opfs.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tree,node_transactions toneBlue
class node_commit,node_batch,node_lsm,node_memtable,node_compaction,node_manifest toneAmber
class node_wal,node_recovery,node_sst,node_vlog,node_files toneMint
class node_checkpoint,node_crypto,node_migration,node_opfs toneRose
class node_user toneIndigo
```

## Components

### API and transactions

- [Tree API](https://github.com/surrealdb/surrealkv/blob/main/src/lib.rs): `src/lib.rs`
- [Transactions](https://github.com/surrealdb/surrealkv/blob/main/src/transaction.rs): `src/transaction.rs`

### LSM engine

- [Commit pipeline](https://github.com/surrealdb/surrealkv/blob/main/src/ring/pipeline.rs): `src/ring/pipeline.rs`
- [Write batches](https://github.com/surrealdb/surrealkv/blob/main/src/batch.rs): `src/batch.rs`
- [LSM coordinator](https://github.com/surrealdb/surrealkv/blob/main/src/lsm.rs): `src/lsm.rs`
- [Memtable](https://github.com/surrealdb/surrealkv/blob/main/src/memtable/mod.rs): `src/memtable/mod.rs`
- [Compaction](https://github.com/surrealdb/surrealkv/blob/main/src/compaction/compactor.rs): `src/compaction/compactor.rs`
- [Levels manifest](https://github.com/surrealdb/surrealkv/blob/main/src/levels/mod.rs): `src/levels/mod.rs`

### Persistence and files

- [Write-ahead log](https://github.com/surrealdb/surrealkv/blob/main/src/wal/manager.rs): `src/wal/manager.rs`
- [WAL recovery](https://github.com/surrealdb/surrealkv/blob/main/src/wal/recovery.rs): `src/wal/recovery.rs`
- [Sorted tables](https://github.com/surrealdb/surrealkv/blob/main/src/sstable/table.rs): `src/sstable/table.rs`
- [Value log](https://github.com/surrealdb/surrealkv/blob/main/src/vlog.rs): `src/vlog.rs`
- [Virtual filesystem](https://github.com/surrealdb/surrealkv/blob/main/src/vfs/mod.rs): `src/vfs/mod.rs`

### Storage capabilities

- [Snapshots and checkpoints](https://github.com/surrealdb/surrealkv/blob/main/src/checkpoint.rs): `src/checkpoint.rs`
- [At-rest encryption](https://github.com/surrealdb/surrealkv/blob/main/src/tde/mod.rs): `src/tde/mod.rs`
- [Format migration](https://github.com/surrealdb/surrealkv/blob/main/src/migrate/mod.rs): `src/migrate/mod.rs`
- [Browser OPFS storage](https://github.com/surrealdb/surrealkv/blob/main/src/storage/opfs.rs): `src/storage/opfs.rs`

### Other components

- **Application**

## Connections

- Application → Tree API: uses
- Tree API → Transactions: offers (evidence: `src/lib.rs`)
- Tree API → LSM coordinator: opens (evidence: `src/lib.rs`)
- Transactions → Commit pipeline: submits commits
- Commit pipeline → Write batches: encodes (evidence: `src/ring/pipeline.rs`)
- Commit pipeline → LSM coordinator: commits (evidence: `src/ring/pipeline.rs`)
- Commit pipeline → Memtable: updates (evidence: `src/ring/pipeline.rs`)
- Commit pipeline → Write-ahead log: appends
- LSM coordinator → Memtable: manages (evidence: `src/lsm.rs`)
- LSM coordinator → Write-ahead log: recovers from (evidence: `src/lsm.rs`)
- LSM coordinator → Compaction: schedules (evidence: `src/lsm.rs`)
- Compaction → Sorted tables: writes tables (evidence: `src/compaction/compactor.rs`)
- Compaction → Levels manifest: updates (evidence: `src/compaction/compactor.rs`)
- LSM coordinator → Levels manifest: uses levels (evidence: `src/lsm.rs`)
- WAL recovery → Write-ahead log: reads records (evidence: `src/wal/recovery.rs`)
- WAL recovery → Memtable: replays batches (evidence: `src/wal/recovery.rs`)
- Memtable → Sorted tables: flushes to (evidence: `src/memtable/mod.rs`)
- Sorted tables → Virtual filesystem: reads and writes (evidence: `src/sstable/table.rs`)
- Write-ahead log → Virtual filesystem: uses filesystem (evidence: `src/wal/manager.rs`)
- Value log → Virtual filesystem: uses filesystem (evidence: `src/vlog.rs`)
- Tree API → Snapshots and checkpoints: exposes (evidence: `src/lib.rs`)
- Snapshots and checkpoints → LSM coordinator: coordinates (evidence: `src/checkpoint.rs`)
- Tree API → At-rest encryption: configures (evidence: `src/lib.rs`)
- LSM coordinator → Format migration: supports migration (evidence: `src/lsm.rs`)
- Virtual filesystem → Browser OPFS storage: selects on browser (evidence: `src/vfs/mod.rs`)
