How helloamj/keyradb works
1. What this repository is
- keyradb is a Go-based, persistent key-value storage engine built around an LSM-tree architecture.
- It is not primarily a web application; the main product is the storage engine, with an HTTP server as the external access layer.
2. Top-level runtime shape
- The executable entrypoint is cmd/server/main.go.
- That server exposes a REST-style API over HTTP and delegates all database behavior to internal/db.
- The rest of the repository is split into focused internal storage components.
3. External interface boundary
- The only consumer-facing runtime boundary is the server in cmd/server.
- Clients interact with keys through HTTP operations like PUT, GET, and DELETE.
- The server does not know about on-disk formats or in-memory structures; it depends only on the db package API.
4. Core orchestrator: internal/db
- internal/db is the engine coordinator that ties together write-ahead logging, in-memory buffering, flushes, compaction, and lookup ordering.
- Its role is to enforce the LSM write/read lifecycle rather than to store data itself.
- The presence of compaction.go and stats.go suggests it also manages background storage maintenance and operational metrics.
5. Write path
- A write goes first to the WAL for durability, then to the memtable for fast in-memory access.
- Once the memtable reaches a configured threshold, the DB flushes it to a new immutable SSTable on disk.
- This gives the system crash recovery via the WAL and fast writes via memory buffering.
6. Read path
- Reads check the memtable first, then older immutable memtables if present, and finally SSTables from newest to oldest.
- This ordering reflects LSM semantics: newer data shadows older data.
- Deletes are likely represented as tombstones in the same lookup chain, so they participate in read resolution and later compaction.
7. internal/wal: durability layer
- The WAL is an append-only log used to recover unflushed writes after a crash or restart.
- It is a separate package, so durability concerns are isolated from indexing and compaction concerns.
- In the architecture diagram, this is the “must succeed first” persistence step before in-memory mutation is trusted.
8. internal/memtable: write buffer
- The memtable is the hot in-memory sorted structure used for recent writes and reads.
- Its implementation is a skip-list-based table, which fits the repository’s emphasis on fast ordered inserts and lookups.
- When full, it becomes immutable and is flushed into persistent storage.
9. internal/sstable: immutable disk storage
- SSTables are the on-disk sorted files that form the persistent storage layer of the LSM tree.
- This package also includes cache.go, indicating an internal caching layer for table access or block reuse.
- SSTables are immutable, which simplifies concurrency and makes compaction possible.
10. internal/bloomfilter and internal/sparseTable: SSTable acceleration
- The Bloom filter is used as a probabilistic “probably not present” check before doing expensive disk reads.
- The sparse table acts as a block-level index, helping locate the right region inside an SSTable without scanning the whole file.
- Together, these two components reduce I/O during reads.
11. Maintenance and lifecycle
- Compaction is part of the DB layer, not the SSTable layer, which matches the LSM responsibility split.
- This is where multiple SSTables are likely merged, obsolete keys removed, and storage rewritten into fewer, denser files.
- stats.go implies runtime accounting for storage size, entry counts, flushes, or compaction state.
12. Package boundary philosophy
- The repository is intentionally modular: WAL, memtable, SSTable, Bloom filter, and sparse index are all separate internal packages.
- That separation is a strong architectural boundary: each package owns one concern and is testable in isolation.
- cmd/server depends upward on db, while db depends downward on the storage primitives.
13. Web assets
- The web/ directory contains a small static frontend (index.html, app.js, style.css).
- It appears to be a dashboard or demo UI, not the primary storage architecture.
- If included in diagrams, it should sit outside the storage engine as an optional client-facing layer.
14. Tooling and validation
- The repository includes unit tests alongside each core package, which reinforces the modular architecture.
- GitHub Actions CI is present, so the engine is intended to be built and tested automatically.
- go.mod shows the project is a standard Go module with no special runtime outside the Go toolchain.
Main entry points: db.go DB Core · sstable.go SSTable · main.go Server · ci.yml CI
Components 15
External API
HTTP entrypoint and consumer-facing boundary.
- Server · Executable entrypoint exposing the key-value API over HTTP.
cmd/server/main.go
LSM Engine
Database coordinator and maintenance logic.
- DB Core · Coordinates writes, reads, flushes, and lifecycle ordering for the LSM tree.
internal/db/db.go - Compaction · Merges SSTables and removes obsolete data during background cleanup.
internal/db/compaction.go - Stats · Tracks runtime storage and maintenance accounting.
internal/db/stats.go
Storage Primitives
Durability, buffering, and on-disk read acceleration.
- WAL · Append-only write-ahead log for crash recovery of unflushed writes.
internal/wal/wal.go - Memtable · Hot in-memory write buffer and first read stop for recent data.
internal/memtable/memtable.go - SSTable · Persistent sorted on-disk table layer for the LSM tree.
internal/sstable/sstable.go - Cache · Reuses SSTable or block data to reduce repeated disk work.
internal/sstable/cache.go - Bloom · Probabilistic negative check before expensive SSTable reads.
internal/bloomfilter/bloomfilter.go - Sparse Index · Locates the right region inside an SSTable without scanning the file.
internal/sparseTable/sparseTable.go
Web UI
Optional browser-facing assets.
Connections 16
- Server → DB Core: calls · HTTP requests are translated into database operations.
- DB Core → WAL: writes first · Durability is enforced through the write-ahead log before in-memory mutation.
- DB Core → Memtable: buffers · New writes are staged in memory after logging.
- DB Core → SSTable: reads / flushes · The DB consults SSTables for lookups and creates them during flushes.
- DB Core → Compaction: coordinates · Background merging and file cleanup is owned by the DB layer.
- DB Core → Stats: tracks · Operational counters and storage state are maintained alongside the coordinator.
- Memtable → SSTable: flushes to · When full, the memtable is turned into an immutable SSTable.
- SSTable → Bloom: checks · Negative lookups are filtered before disk reads.
- SSTable → Sparse Index: indexes · Sparse indexing narrows the on-disk search region.
- SSTable → Cache: uses · Table access is accelerated with caching.
- App JS → Server: requests · The browser client talks to the HTTP server.
- Index → App JS: loads · The HTML shell boots the frontend script.
- Styles → Index: styles · Presentation is separated from markup.
- CI → Server: builds · CI validates the entrypoint and runtime.
- CI → DB Core: tests · CI exercises the core storage engine.
- Module → Server: defines module for · The Go module anchors the server and internal packages.