How dicedb/dice works
Project type and purpose
• DiceDB is an open-source, reactive, in-memory database server written in Go.
• It provides a Redis-like TCP protocol on port 7379, supports real-time subscriptions to data changes, and persists data via AOF/WAL. It also ships with a static documentation site built in Astro.
File structure
• Top‐level “cmd/” and “main.go” – entrypoints for the server binary.
• “internal/server/ironhawk/” – network layer using epoll/kqueue (the Reactor pattern).
• “internal/iomultiplexer/” – cross-platform I/O multiplexing (epoll/kqueue abstractions).
• “internal/cmd/” – individual command implementations (GET, SET, ZADD, etc.).
• “internal/eval/” – evaluation engine (parsing, execution, intermediate data types).
• “internal/store/” – in-memory data structures, eviction, expiration.
• “internal/wal/” and “internal/store/aof.go” – persistence via write‐ahead log and append-only file.
• “internal/shard*” packages – sharding infrastructure for horizontal scaling within a single process.
• “internal/watchmanager/” – observer pattern for reactive query subscriptions.
• “internal/observability/” and “internal/logger/” – metrics, telemetry, logging.
• “docs/” – static site source (Astro framework) for documentation.
• “examples/” – sample client/server apps in Go to demonstrate SDK usage.
Pull additional insights from README
• Emphasizes “fast, reactive, in-memory” – suggesting an event-driven, subscription-based architecture.
• Promotes Docker usage, SDKs, and CLI integration—these are external clients.
Components & relationships
• External Clients (CLI, SDKs, curl)
• TCP Network Layer (internal/server/ironhawk)
• I/O Multiplexer (internal/iomultiplexer)
• Command Parser & Dispatcher (internal/cmd + cmd/root.go)
• Evaluation Engine (internal/eval)
• In-Memory Storage Engine (internal/store)
• Persistence Layer (AOF & WAL in internal/store and internal/wal)
• Shard Manager & Shard Threads (internal/shard*, internal/shardmanager, internal/shardthread)
• Watch Manager (internal/watchmanager)
• Observability & Logging (internal/observability, internal/logger)
• Documentation Site (docs/ – Astro static site generator)
1. Clients → TCP Network Layer
2. Network Layer → I/O Multiplexer → Worker Thread
3. Worker Thread → Command Parser → Dispatcher
4. Dispatcher → Evaluation Engine
5. Evaluation Engine ↔ In-Memory Storage
6. On writes: Storage → Persistence Layer (WAL/AOF)
7. On data changes: Storage → Watch Manager → push notifications → Clients
8. Shard Manager routes commands to Shard Threads for per‐partition processing
9. Observability hooks capture metrics/traces throughout pipeline
c. Architectural patterns & principles:
• Reactor pattern (network I/O multiplexing)
• Sharding (concurrent partitioned in-memory data structures)
• Observer pattern (reactive subscriptions/watch manager)
• Write-ahead logging for durability
• Modular Go packages (separation of concerns, internal packages)
d. Key technologies & libraries:
• Go (modules, go.mod)
• Netpoll: epoll (Linux) & kqueue (Darwin) abstractions
• Protobuf (WAL definitions)
• Zerolog for structured logging
• Astro for static docs site
• Docker, Makefile, Goreleaser for builds & distributions
Detail
• Persistence choose (AOF vs. WAL) and where protobuf fits.
• “I/O Multiplexer” with epoll/kqueue notes.