# pingcap/tikv architecture

> Architecture diagram of the GitHub repository pingcap/tikv, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pingcap/tikv
- Repository: https://github.com/pingcap/tikv

## Overview

This project is TiKV – an open‐source, distributed, transactional key–value database built in Rust with a strong emphasis on consistency and scalability via the Raft consensus algorithm. Both the file structure and the README point to a highly modular system with many components dealing with storage engines (e.g., RocksDB, in-memory engines), replica management via Raft groups, a coprocessor framework for distributed computation, and external integrations (e.g., with the Placement Driver, or PD, and even client drivers).

##### Main Components
- Frontline Client APIs: Interfaces via which TiKV exposes both raw key–value and transactional APIs.
- Distributed Transactional Engine: This includes subcomponents tackling transaction scheduling, concurrency management, snapshot isolation (as seen in the “txn” and “mvcc” directories within the storage subproject).
- Storage Layer: Illustrate how each node (Store) embeds its own storage engine (typically RocksDB) and exposes regions – the basic unit of data distribution. Show that each region is part of a Raft group.
- Raft Consensus Layer: Emphasize the role of Raft for replicating Regions. This is spread out in components such as “raftstore” and “raftstore-v2” that manage log replication, leader election, and cluster configuration.
- Coprocessor Framework: Represent the distributed computing support that allows query execution near the data. This includes both the legacy “coprocessor” layer as well as the “coprocessor_v2” module.
- Placement Driver (PD) Interface: Although PD is a separate project, indicate its importance in cluster metadata, region scheduling, and auto-sharding. The README makes clear that PD and TiKV work in tandem.
- Auxiliary and Supporting Components: Include modules for backup, encryption, resource and error management, performance metrics, logging, and testing frameworks. Many directories in the “components” folder (e.g., backup-stream, cloud, encryption, profiler, etc.) show these roles.

##### Relationships and Interactions
- Data flow: Clients issue key–value requests or transactional commands that are routed to the appropriate TiKV node via a network/API layer.
– Request routing from the client to the relevant Region replica.
– Consensus communication between Raft peers (including leader selection and log replication).
– Periodic reporting and configuration tasks with PD.
– Internal calls between storage management, transaction processing (including the MVCC layer), and coprocessor execution.

##### Architectural Patterns and Design Principles
- Decentralized Data Storage & Consensus: Each node is autonomous in storage management but synchronized via Raft.
- Layered design: At the bottom, emphasize physical storage (RocksDB/in-memory engines) and above that, the transaction, Raft, and API layers.
- Integration points: Mark the extension interfaces such as plugin APIs (coprocessor plugin API) and interfaces for cloud functionalities.

##### Technologies, Frameworks, and Libraries
- Rust as the primary language.
- RocksDB as a key storage engine, along with other engines (in-memory or hybrid).
- Raft as the core consensus mechanism.
- Mention various libraries used for metrics, logging, and error handling (for example, the “clippy” style, test harness, and logging wrappers).

##### Final Assembly
- Draft an initial sketch that slices the entire system into as many components as possible (as indicated by the extensive “components” directory and multiple module folders).

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client("External Client"):::client
    API("Frontline Client APIs"):::api
    Client -->|"sends_request"| API

    TiKV_Node("TiKV Node"):::node
    API -->|"routes"| TiKV_Node

    Txn("Txn"):::transaction
    MVCC("MVCC"):::transaction
    RocksDB("RocksDB"):::engine
    InMemory("In-Memory Engine"):::engine
    Raft1("Raftstore"):::consensus
    Raft2("Raftstore-v2"):::consensus
    Cop1("Coprocessor"):::coprocessor
    Cop2("Coprocessor_v2"):::coprocessor

    TiKV_Node -->|"contains_Txn"| Txn
    TiKV_Node -->|"contains_MVCC"| MVCC
    TiKV_Node -->|"contains_RocksDB"| RocksDB
    TiKV_Node -->|"contains_InMemory"| InMemory
    TiKV_Node -->|"contains_Raft1"| Raft1
    TiKV_Node -->|"contains_Raft2"| Raft2
    TiKV_Node -->|"contains_Coprocessor"| Cop1
    TiKV_Node -->|"contains_Coprocessor_v2"| Cop2

    Txn -->|"uses"| RocksDB
    MVCC -->|"uses"| RocksDB
    Txn -->|"accesses"| InMemory
    MVCC -->|"accesses"| InMemory

    RocksDB -->|"replicated_by"| Raft1
    InMemory -->|"replicated_by"| Raft2

    Txn -->|"executes"| Cop1
    MVCC -->|"executes"| Cop2

    Raft1 -- "sync" --- Raft2

    PD_Interface("PD Interface"):::pd
    TiKV_Node -->|"heartbeat"| PD_Interface

    Auxiliary("Auxiliary Components"):::auxiliary
    TiKV_Node -->|"auxiliary"| Auxiliary
    Backup("Backup-stream"):::auxiliary
    Encryption("Encryption"):::auxiliary
    Profiler("Profiler"):::auxiliary
    Auxiliary -->|"includes"| Backup
    Auxiliary -->|"includes"| Encryption
    Auxiliary -->|"includes"| Profiler

    click API "https://github.com/pingcap/tikv/tree/master/cmd/tikv-server"
    click Txn "https://github.com/pingcap/tikv/tree/master/src/storage/txn"
    click MVCC "https://github.com/pingcap/tikv/tree/master/src/storage/mvcc"
    click RocksDB "https://github.com/pingcap/tikv/tree/master/components/engine_rocks"
    click InMemory "https://github.com/pingcap/tikv/tree/master/components/in_memory_engine"
    click Raft1 "https://github.com/pingcap/tikv/tree/master/components/raftstore"
    click Raft2 "https://github.com/pingcap/tikv/tree/master/components/raftstore-v2"
    click Cop1 "https://github.com/pingcap/tikv/tree/master/src/coprocessor"
    click Cop2 "https://github.com/pingcap/tikv/tree/master/src/coprocessor_v2"
    click PD_Interface "https://github.com/pingcap/tikv/tree/master/components/pd_client"
    click Backup "https://github.com/pingcap/tikv/tree/master/components/backup-stream"
    click Encryption "https://github.com/pingcap/tikv/tree/master/components/encryption"
    click Profiler "https://github.com/pingcap/tikv/tree/master/components/profiler"

    classDef client fill:#cce5ff,stroke:#003366,color:#003366;
    classDef api fill:#d4edda,stroke:#155724,color:#155724;
    classDef node fill:#e0e0e0,stroke:#666666,color:#333333;
    classDef transaction fill:#c3e6cb,stroke:#155724,color:#155724;
    classDef engine fill:#fff3cd,stroke:#856404,color:#856404;
    classDef consensus fill:#f8d7da,stroke:#721c24,color:#721c24;
    classDef coprocessor fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
    classDef pd fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef auxiliary fill:#f5c6cb,stroke:#721c24,color:#721c24;
```
