How snapchat/keydb works
Project Type and Purpose
• KeyDB is a high-performance, Redis-compatible, in-memory key–value store.
• It is designed with multithreading in mind (each new client connection gets assigned to a worker thread), active replication, persistence (RDB/AOF), and optional flash storage support while retaining full compatibility with Redis’ protocol and modules.
• The README clearly emphasizes its positioning as a drop–in replacement for Redis with added performance features and extended configuration options.
File Structure
• The file tree is large and contains a traditional “src” directory with the production code including core engine files (for example, files like server.cpp, cluster.cpp, dict.c, modules folder, replication-related source files, TLS support, and persistence implementations).
• There are directories for external dependencies (“deps”) such as jemalloc, hiredis, lua, and other libraries that enhance memory management and extend functionality.
• The “tests”, “pkg”, and “docs” components indicate careful testing, packaging scripts and infrastructure for building/packaging (including Dockerfiles and systemd scripts).
• The extensive “deps” tree shows that KeyDB integrates with several third–party libraries, while the “src/modules” folder indicates a clear plugin or modules extension system.
• Build files such as Makefile, build.yaml, and ci workflows show continuous integration and configurable build targets (including choices for TLS, flash, allocator variants, etc.).
Main Components and Their Relationships
a. External Clients and API Layer
– Represent external clients connecting via the Redis protocol. These connections may use TLS if enabled.
– Show both CLI (keydb-cli) and programmatic clients.
b. Network Input/Dispatcher Layer
– A connection acceptor (using SO_REUSEPORT) listens for incoming TCP connections.
– Each accepted connection is dispatched to an available worker thread.
– Mark load-balancing via the network (including support for active client balancing and replica–weighting as configured).
c. Worker Thread Pool and Event Loops
– Illustrate multiple threads running concurrent “event loops” (see files such as ae.cpp, ae_epoll.cpp, and similar implementations).
– Each thread is responsible for query parsing, non–blocking IO, and assignment of tasks.
d. Shared Data Store and Core Engine
– Emphasize that access to the data store is guarded by spinlocks (to maintain low contention across threads).
– Include support for transactions (lock held during EXEC) with concurrency control.
e. Persistence and Storage Engine
– Illustrate a persistence sub–component responsible for RDB snapshots, AOF logging, and the optional direct dump/load to/from AWS S3.
– Additionally, note “FLASH storage” support which directs data to SSD volumes, with experimental status.
f. Replication and Active–Active Cluster Manager
– Show a replication component responsible for active replication, multi–master setups, and replica load balancing.
– Indicate bi–directional communication between KeyDB instances for active replication.
g. Modules and Extension Points
– Include a modules layer (from src/modules) that provides the extension API.
– Modules work with a global interpreter lock (GIL), pausing threads as needed, and are integrated into the main engine.
h. Auxiliary and Configuration Components
– Add a block for configuration parsing (from keydb.conf, command–line parameters, and build–time options).
– Mention external dependencies (deps such as hiredis for Redis protocol communication, Jemalloc for memory management, Lua for scripting, etc.).
Architectural Patterns and Design Principles
• Multithreading – clearly show separation between dispatcher and worker threads to reflect KeyDB’s strategy for parallelizing network I/O and query processing.
• Layered Architecture – highlight the separation of concerns (client interface, request processing, data storage, persistence, replication).
• Modular Design – capture points where plugins or external modules (potentially JavaScript via ModJS or native modules) hook into the flow.
• Concurrency Control – feature the synchronization (via spinlocks) in the data store access.
• Extensibility & Compatibility – show how KeyDB remains a superset of Redis, supporting all Redis modules and protocols.