# snapchat/keydb architecture

> Architecture diagram of the GitHub repository snapchat/keydb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snapchat/keydb
- Repository: https://github.com/snapchat/keydb

## Overview

##### Project Type and Purpose
- KeyDB is a high-performance, Redis-compatible, in-memory key–value store.
- It is designed with multithreading in mind (each new client connection gets assigned to a worker thread), active replication, persistence (RDB/AOF), and optional flash storage support while retaining full compatibility with Redis’ protocol and modules.
- The README clearly emphasizes its positioning as a drop–in replacement for Redis with added performance features and extended configuration options.

##### File Structure
- The file tree is large and contains a traditional “src” directory with the production code including core engine files (for example, files like server.cpp, cluster.cpp, dict.c, modules folder, replication-related source files, TLS support, and persistence implementations).
- There are directories for external dependencies (“deps”) such as jemalloc, hiredis, lua, and other libraries that enhance memory management and extend functionality.
- The “tests”, “pkg”, and “docs” components indicate careful testing, packaging scripts and infrastructure for building/packaging (including Dockerfiles and systemd scripts).
- The extensive “deps” tree shows that KeyDB integrates with several third–party libraries, while the “src/modules” folder indicates a clear plugin or modules extension system.
- Build files such as Makefile, build.yaml, and ci workflows show continuous integration and configurable build targets (including choices for TLS, flash, allocator variants, etc.).

##### Main Components and Their Relationships
a. External Clients and API Layer
– Represent external clients connecting via the Redis protocol. These connections may use TLS if enabled.
– Show both CLI (keydb-cli) and programmatic clients.
b. Network Input/Dispatcher Layer
– A connection acceptor (using SO_REUSEPORT) listens for incoming TCP connections.
– Each accepted connection is dispatched to an available worker thread.
– Mark load-balancing via the network (including support for active client balancing and replica–weighting as configured).
c. Worker Thread Pool and Event Loops
– Illustrate multiple threads running concurrent “event loops” (see files such as ae.cpp, ae_epoll.cpp, and similar implementations).
– Each thread is responsible for query parsing, non–blocking IO, and assignment of tasks.
d. Shared Data Store and Core Engine
– Emphasize that access to the data store is guarded by spinlocks (to maintain low contention across threads).
– Include support for transactions (lock held during EXEC) with concurrency control.
e. Persistence and Storage Engine
– Illustrate a persistence sub–component responsible for RDB snapshots, AOF logging, and the optional direct dump/load to/from AWS S3.
– Additionally, note “FLASH storage” support which directs data to SSD volumes, with experimental status.
f. Replication and Active–Active Cluster Manager
– Show a replication component responsible for active replication, multi–master setups, and replica load balancing.
– Indicate bi–directional communication between KeyDB instances for active replication.
g. Modules and Extension Points
– Include a modules layer (from src/modules) that provides the extension API.
– Modules work with a global interpreter lock (GIL), pausing threads as needed, and are integrated into the main engine.
h. Auxiliary and Configuration Components
– Add a block for configuration parsing (from keydb.conf, command–line parameters, and build–time options).
– Mention external dependencies (deps such as hiredis for Redis protocol communication, Jemalloc for memory management, Lua for scripting, etc.).

##### Architectural Patterns and Design Principles
- Multithreading – clearly show separation between dispatcher and worker threads to reflect KeyDB’s strategy for parallelizing network I/O and query processing.
- Layered Architecture – highlight the separation of concerns (client interface, request processing, data storage, persistence, replication).
- Modular Design – capture points where plugins or external modules (potentially JavaScript via ModJS or native modules) hook into the flow.
- Concurrency Control – feature the synchronization (via spinlocks) in the data store access.
- Extensibility & Compatibility – show how KeyDB remains a superset of Redis, supporting all Redis modules and protocols.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        A["External Clients & API"]:::client
    end

    %% Network Input/Dispatcher Layer
    subgraph "Network Layer"
        B["Network Dispatcher"]:::network
    end

    %% Worker Thread Pool and Event Loops
    subgraph "Worker Threads"
        C1["Event Loop (ae.cpp)"]:::worker
        C2["Event Loop (ae_epoll.cpp)"]:::worker
    end

    %% Shared Data Store and Core Engine
    subgraph "Core Engine"
        D1["Shared Data Store (dict.cpp)"]:::core
        D2["Database Engine (db.cpp)"]:::core
    end

    %% Persistence and Storage Engine
    subgraph "Persistence Engine"
        E1["AOF Persistence (aof.cpp)"]:::persistence
        E2["RDB Snapshots (rdb.cpp)"]:::persistence
    end

    %% Replication and Active–Active Cluster Manager
    subgraph "Replication Manager"
        F1["Active Replication (replication.cpp)"]:::replication
        F2["Cluster Management (cluster.cpp)"]:::replication
    end

    %% Modules and Extension Points
    subgraph "Modules API"
        G["Modules Extension API"]:::modules
    end

    %% Auxiliary and Configuration Components
    subgraph "Configuration"
        H1["Config File (keydb.conf)"]:::configuration
        H2["Config Code (config.cpp & config.h)"]:::configuration
    end

    %% Connections
    A -->|"connects"| B
    B -->|"dispatches"| C1
    B -->|"dispatches"| C2
    C1 -->|"processes"| D1
    C1 -->|"processes"| D2
    C2 -->|"processes"| D1
    C2 -->|"processes"| D2
    D1 -->|"triggers"| E1
    D1 -->|"triggers"| E2
    D2 -->|"triggers"| E1
    D2 -->|"triggers"| E2
    D1 -->|"updates"| F1
    D1 -->|"updates"| F2
    D2 -->|"updates"| F1
    D2 -->|"updates"| F2
    D1 -->|"invokes"| G
    D2 -->|"invokes"| G
    H2 --- B
    H2 --- G
    H1 --- H2

    %% Click Events
    click A "https://github.com/snapchat/keydb/blob/main/src/redis-cli.c"
    click B "https://github.com/snapchat/keydb/blob/main/src/anet.c"
    click B "https://github.com/snapchat/keydb/blob/main/src/server.cpp"
    click C1 "https://github.com/snapchat/keydb/blob/main/src/ae.cpp"
    click C2 "https://github.com/snapchat/keydb/blob/main/src/ae_epoll.cpp"
    click D1 "https://github.com/snapchat/keydb/blob/main/src/dict.cpp"
    click D2 "https://github.com/snapchat/keydb/blob/main/src/db.cpp"
    click E1 "https://github.com/snapchat/keydb/blob/main/src/aof.cpp"
    click E2 "https://github.com/snapchat/keydb/blob/main/src/rdb.cpp"
    click F1 "https://github.com/snapchat/keydb/blob/main/src/replication.cpp"
    click F2 "https://github.com/snapchat/keydb/blob/main/src/cluster.cpp"
    click G "https://github.com/snapchat/keydb/tree/main/src/modules"
    click H1 "https://github.com/snapchat/keydb/blob/main/keydb.conf"
    click H2 "https://github.com/snapchat/keydb/blob/main/src/config.cpp"
    click H2 "https://github.com/snapchat/keydb/blob/main/src/config.h"

    %% Styles
    classDef client fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef network fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef worker fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef core fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef persistence fill:#87CEEB,stroke:#333,stroke-width:2px;
    classDef replication fill:#DDA0DD,stroke:#333,stroke-width:2px;
    classDef modules fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef configuration fill:#F0E68C,stroke:#333,stroke-width:2px;
```
