# helloamj/keyradb architecture

> Architecture diagram of the GitHub repository helloamj/keyradb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/helloamj/keyradb
- Repository: https://github.com/helloamj/keyradb

## Overview

1. **What this repository is**
   - `keyradb` is a Go-based, persistent key-value storage engine built around an **LSM-tree** architecture.
   - It is not primarily a web application; the main product is the storage engine, with an HTTP server as the external access layer.

2. **Top-level runtime shape**
   - The executable entrypoint is `cmd/server/main.go`.
   - That server exposes a REST-style API over HTTP and delegates all database behavior to `internal/db`.
   - The rest of the repository is split into focused internal storage components.

3. **External interface boundary**
   - The only consumer-facing runtime boundary is the server in `cmd/server`.
   - Clients interact with keys through HTTP operations like `PUT`, `GET`, and `DELETE`.
   - The server does not know about on-disk formats or in-memory structures; it depends only on the `db` package API.

4. **Core orchestrator: `internal/db`**
   - `internal/db` is the engine coordinator that ties together write-ahead logging, in-memory buffering, flushes, compaction, and lookup ordering.
   - Its role is to enforce the LSM write/read lifecycle rather than to store data itself.
   - The presence of `compaction.go` and `stats.go` suggests it also manages background storage maintenance and operational metrics.

5. **Write path**
   - A write goes first to the **WAL** for durability, then to the **memtable** for fast in-memory access.
   - Once the memtable reaches a configured threshold, the DB flushes it to a new immutable SSTable on disk.
   - This gives the system crash recovery via the WAL and fast writes via memory buffering.

6. **Read path**
   - Reads check the **memtable** first, then older immutable memtables if present, and finally SSTables from newest to oldest.
   - This ordering reflects LSM semantics: newer data shadows older data.
   - Deletes are likely represented as tombstones in the same lookup chain, so they participate in read resolution and later compaction.

7. **`internal/wal`: durability layer**
   - The WAL is an append-only log used to recover unflushed writes after a crash or restart.
   - It is a separate package, so durability concerns are isolated from indexing and compaction concerns.
   - In the architecture diagram, this is the “must succeed first” persistence step before in-memory mutation is trusted.

8. **`internal/memtable`: write buffer**
   - The memtable is the hot in-memory sorted structure used for recent writes and reads.
   - Its implementation is a skip-list-based table, which fits the repository’s emphasis on fast ordered inserts and lookups.
   - When full, it becomes immutable and is flushed into persistent storage.

9. **`internal/sstable`: immutable disk storage**
   - SSTables are the on-disk sorted files that form the persistent storage layer of the LSM tree.
   - This package also includes `cache.go`, indicating an internal caching layer for table access or block reuse.
   - SSTables are immutable, which simplifies concurrency and makes compaction possible.

10. **`internal/bloomfilter` and `internal/sparseTable`: SSTable acceleration**
   - The Bloom filter is used as a probabilistic “probably not present” check before doing expensive disk reads.
   - The sparse table acts as a block-level index, helping locate the right region inside an SSTable without scanning the whole file.
   - Together, these two components reduce I/O during reads.

11. **Maintenance and lifecycle**
   - Compaction is part of the DB layer, not the SSTable layer, which matches the LSM responsibility split.
   - This is where multiple SSTables are likely merged, obsolete keys removed, and storage rewritten into fewer, denser files.
   - `stats.go` implies runtime accounting for storage size, entry counts, flushes, or compaction state.

12. **Package boundary philosophy**
   - The repository is intentionally modular: WAL, memtable, SSTable, Bloom filter, and sparse index are all separate internal packages.
   - That separation is a strong architectural boundary: each package owns one concern and is testable in isolation.
   - `cmd/server` depends upward on `db`, while `db` depends downward on the storage primitives.

13. **Web assets**
   - The `web/` directory contains a small static frontend (`index.html`, `app.js`, `style.css`).
   - It appears to be a dashboard or demo UI, not the primary storage architecture.
   - If included in diagrams, it should sit outside the storage engine as an optional client-facing layer.

14. **Tooling and validation**
   - The repository includes unit tests alongside each core package, which reinforces the modular architecture.
   - GitHub Actions CI is present, so the engine is intended to be built and tested automatically.
   - `go.mod` shows the project is a standard Go module with no special runtime outside the Go toolchain.

Main entry points: [`internal/db/db.go`](https://github.com/helloamj/keyradb/blob/main/internal/db/db.go) (DB Core), [`internal/sstable/sstable.go`](https://github.com/helloamj/keyradb/blob/main/internal/sstable/sstable.go) (SSTable), [`cmd/server/main.go`](https://github.com/helloamj/keyradb/blob/main/cmd/server/main.go) (Server), [`.github/workflows/ci.yml`](https://github.com/helloamj/keyradb/blob/main/.github/workflows/ci.yml) (CI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["External API"]
  node_cmd_server_main["Server<br/>http entry<br/>[main.go]"]
end

subgraph group_engine["LSM Engine"]
  node_db_core["DB Core<br/>lsm coordinator<br/>[db.go]"]
  node_db_compaction["Compaction<br/>maintenance<br/>[compaction.go]"]
  node_db_stats["Stats<br/>metrics<br/>[stats.go]"]
end

subgraph group_storage["Storage Primitives"]
  node_wal[("WAL<br/>durable log<br/>[wal.go]")]
  node_memtable["Memtable<br/>in-memory buffer<br/>[memtable.go]"]
  node_sstable[("SSTable<br/>immutable files<br/>[sstable.go]")]
  node_sstable_cache["Cache<br/>[cache.go]"]
  node_bloomfilter(("Bloom<br/>presence filter<br/>[bloomfilter.go]"))
  node_sparse_table["Sparse Index<br/>block index<br/>[sparseTable.go]"]
end

subgraph group_ui["Web UI"]
  node_web_app["App JS<br/>frontend client<br/>[app.js]"]
  node_web_index["Index<br/>static page<br/>[index.html]"]
  node_web_style["Styles<br/>presentation<br/>[style.css]"]
end

subgraph group_tooling["Tooling"]
  node_ci["CI<br/>workflow<br/>[ci.yml]"]
  node_gomod["Module<br/>[go.mod]"]
end

node_cmd_server_main -->|"calls"| node_db_core
node_db_core -->|"writes first"| node_wal
node_db_core -->|"buffers"| node_memtable
node_db_core -->|"reads / flushes"| node_sstable
node_db_core -->|"coordinates"| node_db_compaction
node_db_core -->|"tracks"| node_db_stats
node_memtable -->|"flushes to"| node_sstable
node_sstable -->|"checks"| node_bloomfilter
node_sstable -->|"indexes"| node_sparse_table
node_sstable -->|"uses"| node_sstable_cache
node_web_app -->|"requests"| node_cmd_server_main
node_web_index -->|"loads"| node_web_app
node_web_style -->|"styles"| node_web_index
node_ci -->|"builds"| node_cmd_server_main
node_ci -->|"tests"| node_db_core
node_gomod -->|"defines module for"| node_cmd_server_main

click node_cmd_server_main "https://github.com/helloamj/keyradb/blob/main/cmd/server/main.go"
click node_db_core "https://github.com/helloamj/keyradb/blob/main/internal/db/db.go"
click node_db_compaction "https://github.com/helloamj/keyradb/blob/main/internal/db/compaction.go"
click node_db_stats "https://github.com/helloamj/keyradb/blob/main/internal/db/stats.go"
click node_wal "https://github.com/helloamj/keyradb/blob/main/internal/wal/wal.go"
click node_memtable "https://github.com/helloamj/keyradb/blob/main/internal/memtable/memtable.go"
click node_sstable "https://github.com/helloamj/keyradb/blob/main/internal/sstable/sstable.go"
click node_sstable_cache "https://github.com/helloamj/keyradb/blob/main/internal/sstable/cache.go"
click node_bloomfilter "https://github.com/helloamj/keyradb/blob/main/internal/bloomfilter/bloomfilter.go"
click node_sparse_table "https://github.com/helloamj/keyradb/blob/main/internal/sparseTable/sparseTable.go"
click node_web_app "https://github.com/helloamj/keyradb/blob/main/web/app.js"
click node_web_index "https://github.com/helloamj/keyradb/blob/main/web/index.html"
click node_web_style "https://github.com/helloamj/keyradb/blob/main/web/style.css"
click node_ci "https://github.com/helloamj/keyradb/blob/main/.github/workflows/ci.yml"
click node_gomod "https://github.com/helloamj/keyradb/blob/main/go.mod"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cmd_server_main toneBlue
class node_db_core,node_db_compaction,node_db_stats toneAmber
class node_wal,node_memtable,node_sstable,node_sstable_cache,node_bloomfilter,node_sparse_table toneMint
class node_web_app,node_web_index,node_web_style toneRose
class node_ci,node_gomod toneIndigo
```

## Components

### External API

HTTP entrypoint and consumer-facing boundary.

- [Server](https://github.com/helloamj/keyradb/blob/main/cmd/server/main.go): http entry · Executable entrypoint exposing the key-value API over HTTP. · `cmd/server/main.go`

### LSM Engine

Database coordinator and maintenance logic.

- [DB Core](https://github.com/helloamj/keyradb/blob/main/internal/db/db.go): lsm coordinator · Coordinates writes, reads, flushes, and lifecycle ordering for the LSM tree. · `internal/db/db.go`
- [Compaction](https://github.com/helloamj/keyradb/blob/main/internal/db/compaction.go): maintenance · Merges SSTables and removes obsolete data during background cleanup. · `internal/db/compaction.go`
- [Stats](https://github.com/helloamj/keyradb/blob/main/internal/db/stats.go): metrics · Tracks runtime storage and maintenance accounting. · `internal/db/stats.go`

### Storage Primitives

Durability, buffering, and on-disk read acceleration.

- [WAL](https://github.com/helloamj/keyradb/blob/main/internal/wal/wal.go): durable log · Append-only write-ahead log for crash recovery of unflushed writes. · `internal/wal/wal.go`
- [Memtable](https://github.com/helloamj/keyradb/blob/main/internal/memtable/memtable.go): in-memory buffer · Hot in-memory write buffer and first read stop for recent data. · `internal/memtable/memtable.go`
- [SSTable](https://github.com/helloamj/keyradb/blob/main/internal/sstable/sstable.go): immutable files · Persistent sorted on-disk table layer for the LSM tree. · `internal/sstable/sstable.go`
- [Cache](https://github.com/helloamj/keyradb/blob/main/internal/sstable/cache.go): table cache · Reuses SSTable or block data to reduce repeated disk work. · `internal/sstable/cache.go`
- [Bloom](https://github.com/helloamj/keyradb/blob/main/internal/bloomfilter/bloomfilter.go): presence filter · Probabilistic negative check before expensive SSTable reads. · `internal/bloomfilter/bloomfilter.go`
- [Sparse Index](https://github.com/helloamj/keyradb/blob/main/internal/sparseTable/sparseTable.go): block index · Locates the right region inside an SSTable without scanning the file. · `internal/sparseTable/sparseTable.go`

### Web UI

Optional browser-facing assets.

- [App JS](https://github.com/helloamj/keyradb/blob/main/web/app.js): frontend client · Browser-side client logic for the optional web interface. · `web/app.js`
- [Index](https://github.com/helloamj/keyradb/blob/main/web/index.html): static page · HTML shell for the optional browser UI. · `web/index.html`
- [Styles](https://github.com/helloamj/keyradb/blob/main/web/style.css): presentation · Visual styling for the optional web UI. · `web/style.css`

### Tooling

Build and validation infrastructure.

- [CI](https://github.com/helloamj/keyradb/blob/main/.github/workflows/ci.yml): workflow · GitHub Actions build and test pipeline. · `.github/workflows/ci.yml`
- [Module](https://github.com/helloamj/keyradb/blob/main/go.mod): go module · Go module boundary and dependency definition. · `go.mod`

## Connections

- Server → DB Core: calls · HTTP requests are translated into database operations.
- DB Core → WAL: writes first · Durability is enforced through the write-ahead log before in-memory mutation.
- DB Core → Memtable: buffers · New writes are staged in memory after logging.
- DB Core → SSTable: reads / flushes · The DB consults SSTables for lookups and creates them during flushes.
- DB Core → Compaction: coordinates · Background merging and file cleanup is owned by the DB layer.
- DB Core → Stats: tracks · Operational counters and storage state are maintained alongside the coordinator.
- Memtable → SSTable: flushes to · When full, the memtable is turned into an immutable SSTable.
- SSTable → Bloom: checks · Negative lookups are filtered before disk reads.
- SSTable → Sparse Index: indexes · Sparse indexing narrows the on-disk search region.
- SSTable → Cache: uses · Table access is accelerated with caching.
- App JS → Server: requests · The browser client talks to the HTTP server.
- Index → App JS: loads · The HTML shell boots the frontend script.
- Styles → Index: styles · Presentation is separated from markup.
- CI → Server: builds · CI validates the entrypoint and runtime.
- CI → DB Core: tests · CI exercises the core storage engine.
- Module → Server: defines module for · The Go module anchors the server and internal packages.