# databendlabs/openraft architecture

> Architecture diagram of the GitHub repository databendlabs/openraft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databendlabs/openraft
- Repository: https://github.com/databendlabs/openraft

## Overview

##### Project Type and Purpose
- This is an open-source Rust library/crate (“openraft”) implementing the Raft consensus protocol as an embeddable engine.
- It offers a pluggable, asynchronous, event-driven Raft core (leader election, log replication, snapshotting, dynamic membership) and exposes trait-based extension points for storage, networking, and state-machine persistence.
- Example applications (under `examples/`) wire the core to in-memory stores, gRPC transport, RocksDB, Opendal snapshots, single-threaded loops, etc.
- A benchmarking harness (in `cluster_benchmark/`) and end-to-end tests (`tests/`) validate correctness and performance.

##### High-Level Components
a. Raft API Layer
- The public `Raft<T>` struct drives the node’s lifecycle and exposes methods like `trigger_elect()`, `apply()`, `add_learner()`, `trigger_snapshot()`, `ensure_linearizable()`.
- Configured via a `Config` object specifying timeouts, feature flags (heartbeat/elect toggles), and tracing levels.
b. Core Engine
- Submodules in `openraft/src/core/` and `openraft/src/engine/`:
- Event loop (“engine_impl”) dispatching Election, Heartbeat, Replication, Command, Snapshot, Vote handlers.
- State machines managing `ServerState`, `ReplicateState`, `MembershipState`.
- Queueing & batching of Raft messages for throughput.
c. Networking Abstraction
- The `RaftNetwork` trait in `openraft/src/network/` defines RPCs: AppendEntries, Vote, InstallSnapshot.
- Two versions (v1, v2) with adapter modules supporting trait upgrades.
- Examples implement this trait over gRPC (`examples/raft-kv-memstore-grpc`), simple TCP, or in-process channels.
d. Storage Abstraction
- The `RaftLogStorage` and snapshot traits under `openraft/src/storage/`.
- Responsibilities: persisting logs, state machine snapshots, on-disk indexing.
- Examples and `stores/` show in-memory and RocksDB implementations.
e. State-Machine Hooks
- The `RaftStateMachine` trait applied to user data upon commit.
- Serialized commands carried in `openraft/src/entry/`.
f. Supporting Subsystems
- Type configuration (`type_config/`) to swap Tokio or monoio runtimes, channel/backpressure implementations.
- Tracing instrumentation via `tracing` crate.
- Metrics collector in `openraft/src/metrics/`.
- Testing utilities under `openraft/src/testing/`.
g. Examples & Benchmarks
- Multiple sample apps, each wiring network, storage, and SM to the core.
- `cluster_benchmark/` runs high-throughput put workloads.

##### Component Interactions and Data Flow
1. Application invokes `Raft::new(...)` with trait implementations for network, storage, and SM.
2. The Raft core starts an async task:
– Election module monitors timeouts or manual triggers → broadcasts `RequestVote` RPCs via `RaftNetwork`.
– Leader on election success handles client writes → appends to log storage, replicates via `AppendEntries` RPCs.
– Followers receive RPCs, persist to storage, reply, and signal commit.
– Commit notification passed to `RaftStateMachine` to apply entries.
– Snapshot logic offloads old logs into snapshot storage and replicates via `InstallSnapshot` RPCs.
3. All subsystems emit metrics and tracing spans.
4. Type-config layers manage task spawning, channel backpressure, and runtime selection.

##### Architectural Patterns & Principles
- Event-driven, non-blocking Raft state machine using Tokio (async/await).
- Strict separation of core (protocol) and pluggable interfaces (network/storage/SM) via Rust traits.
- Module organization by feature area (config, core, engine, network, storage, replication, membership, testing).
- Examples illustrate dependency-inversion: the core depends only on trait interfaces.

##### Key Technologies & Libraries
- Rust async ecosystem: Tokio (or alternative runtimes via `type_config`), futures, channels.
- Traits for `RaftLogStorage`, `RaftNetwork`, `RaftStateMachine`.
- `tracing` for instrumentation, `metrics` for runtime stats.
- Example transports: gRPC (via `tonic`), Opendal, RocksDB, in-memory.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Application and Examples
    subgraph "Applications / Examples" 
        direction TB
        App["Client Application"]:::examples
        Ex1["raft-kv-memstore-grpc"]:::examples
        Ex2["raft-kv-memstore-network-v2"]:::examples
        Ex3["raft-kv-memstore-opendal-snapshot-data"]:::examples
        Ex4["raft-kv-memstore-singlethreaded"]:::examples
    end

    %% Raft API Layer
    subgraph "Raft API Layer"
        direction TB
        RaftLib["lib.rs"]:::core-protocol
        RaftMod["raft/mod.rs"]:::core-protocol
        RaftNode["node.rs"]:::core-protocol
    end

    %% Configuration
    subgraph "Configuration"
        direction TB
        ConfigMod["config/mod.rs"]:::config
        ConfigObj["config/config.rs"]:::config
    end

    %% Core Engine
    subgraph "Core Engine"
        direction TB
        EngineImpl["engine_impl.rs"]:::core-protocol
        Handlers["handler/"]:::core-protocol
        EngineOut["engine_output.rs"]:::core-protocol
    end

    %% Core State Machines
    subgraph "Core State Machines"
        direction TB
        ServerState["server_state.rs"]:::core-protocol
        ReplicateState["replication_state.rs"]:::core-protocol
        HeartbeatSM["heartbeat/"]:::core-protocol
        MembershipSM["sm/"]:::core-protocol
    end

    %% Networking Abstraction
    subgraph "Networking Interface"
        direction TB
        NetMod["network/mod.rs"]:::interface
        NetV1["v1/network.rs"]:::implementation
        NetV2["v2/network.rs"]:::implementation
    end

    %% Storage Abstraction
    subgraph "Storage Interface"
        direction TB
        StorageMod["storage/mod.rs"]:::interface
        LogStorage["v2/raft_log_storage.rs"]:::implementation
        SnapshotBuilder["v2/raft_snapshot_builder.rs"]:::implementation
    end

    %% State-Machine Hooks
    subgraph "State Machine Hooks"
        direction TB
        SMTraits["traits.rs"]:::interface
        SMPayload["payload.rs"]:::implementation
        SMCommand["sm/command.rs"]:::implementation
    end

    %% Type Configuration / Runtime
    subgraph "Type Config / Runtime"
        direction TB
        TypeConfig["type_config/mod.rs"]:::config
        TokioRuntime["async_runtime/tokio_impls/tokio_runtime.rs"]:::implementation
    end

    %% Metrics & Tracing
    subgraph "Metrics & Tracing"
        direction TB
        MetricsMod["metrics/mod.rs"]:::interface
        RaftMetrics["metrics/raft_metrics.rs"]:::implementation
    end

    %% Testing Utilities
    subgraph "Testing Utilities"
        direction TB
        TestingCommon["testing/common.rs"]:::implementation
        StoreBuilder["testing/log/store_builder.rs"]:::implementation
    end

    %% Benchmark Harness
    Bench["cluster_benchmark/"]:::examples

    %% Tests
    subgraph "End-to-End Tests"
        direction TB
        TestMain["tests/src/main.rs"]:::examples
        TestSuite["tests/tests/"]:::examples
    end

    %% Connections
    App -->|uses RPC| RaftLib
    Ex1 --> RaftLib
    Ex2 --> RaftLib
    Ex3 --> RaftLib
    Ex4 --> RaftLib

    RaftLib --> RaftMod
    RaftMod --> RaftNode
    RaftNode --> ConfigObj
    RaftNode --> EngineImpl

    EngineImpl --> Handlers
    EngineImpl --> EngineOut
    EngineImpl --> ServerState
    EngineImpl --> ReplicateState
    EngineImpl --> HeartbeatSM
    EngineImpl --> MembershipSM

    EngineImpl -->|calls RPC| NetMod
    NetMod --> NetV1
    NetMod --> NetV2

    EngineImpl -->|persists| StorageMod
    StorageMod --> LogStorage
    StorageMod --> SnapshotBuilder

    EngineImpl -->|applies| SMTraits
    SMTraits --> SMPayload
    SMTraits --> SMCommand

    EngineImpl -->|spawns tasks| TypeConfig
    TypeConfig --> TokioRuntime

    EngineImpl -->|emits| MetricsMod
    MetricsMod --> RaftMetrics

    RaftLib --> TestingCommon
    TestingCommon --> StoreBuilder

    RaftLib --> Bench
    RaftLib --> TestMain
    TestMain --> TestSuite

    %% Click Events
    click RaftLib "https://github.com/databendlabs/openraft/blob/main/openraft/src/lib.rs"
    click RaftMod "https://github.com/databendlabs/openraft/blob/main/openraft/src/raft/mod.rs"
    click RaftNode "https://github.com/databendlabs/openraft/blob/main/openraft/src/node.rs"
    click ConfigMod "https://github.com/databendlabs/openraft/blob/main/openraft/src/config/mod.rs"
    click ConfigObj "https://github.com/databendlabs/openraft/blob/main/openraft/src/config/config.rs"
    click EngineImpl "https://github.com/databendlabs/openraft/blob/main/openraft/src/engine/engine_impl.rs"
    click Handlers "https://github.com/databendlabs/openraft/tree/main/openraft/src/engine/handler/"
    click EngineOut "https://github.com/databendlabs/openraft/blob/main/openraft/src/engine/engine_output.rs"
    click ServerState "https://github.com/databendlabs/openraft/blob/main/openraft/src/core/server_state.rs"
    click ReplicateState "https://github.com/databendlabs/openraft/blob/main/openraft/src/core/replication_state.rs"
    click HeartbeatSM "https://github.com/databendlabs/openraft/tree/main/openraft/src/core/heartbeat/"
    click MembershipSM "https://github.com/databendlabs/openraft/tree/main/openraft/src/core/sm/"
    click NetMod "https://github.com/databendlabs/openraft/blob/main/openraft/src/network/mod.rs"
    click NetV1 "https://github.com/databendlabs/openraft/blob/main/openraft/src/network/v1/network.rs"
    click NetV2 "https://github.com/databendlabs/openraft/blob/main/openraft/src/network/v2/network.rs"
    click StorageMod "https://github.com/databendlabs/openraft/blob/main/openraft/src/storage/mod.rs"
    click LogStorage "https://github.com/databendlabs/openraft/blob/main/openraft/src/storage/v2/raft_log_storage.rs"
    click SnapshotBuilder "https://github.com/databendlabs/openraft/blob/main/openraft/src/storage/v2/raft_snapshot_builder.rs"
    click SMTraits "https://github.com/databendlabs/openraft/blob/main/openraft/src/entry/traits.rs"
    click SMPayload "https://github.com/databendlabs/openraft/blob/main/openraft/src/entry/payload.rs"
    click SMCommand "https://github.com/databendlabs/openraft/blob/main/openraft/src/core/sm/command.rs"
    click TypeConfig "https://github.com/databendlabs/openraft/blob/main/openraft/src/type_config/mod.rs"
    click TokioRuntime "https://github.com/databendlabs/openraft/blob/main/openraft/src/type_config/async_runtime/tokio_impls/tokio_runtime.rs"
    click MetricsMod "https://github.com/databendlabs/openraft/blob/main/openraft/src/metrics/mod.rs"
    click RaftMetrics "https://github.com/databendlabs/openraft/blob/main/openraft/src/metrics/raft_metrics.rs"
    click TestingCommon "https://github.com/databendlabs/openraft/blob/main/openraft/src/testing/common.rs"
    click StoreBuilder "https://github.com/databendlabs/openraft/blob/main/openraft/src/testing/log/store_builder.rs"
    click Ex1 "https://github.com/databendlabs/openraft/tree/main/examples/raft-kv-memstore-grpc/"
    click Ex2 "https://github.com/databendlabs/openraft/tree/main/examples/raft-kv-memstore-network-v2/"
    click Ex3 "https://github.com/databendlabs/openraft/tree/main/examples/raft-kv-memstore-opendal-snapshot-data/"
    click Ex4 "https://github.com/databendlabs/openraft/tree/main/examples/raft-kv-memstore-singlethreaded/"
    click Bench "https://github.com/databendlabs/openraft/tree/main/cluster_benchmark/"
    click TestMain "https://github.com/databendlabs/openraft/blob/main/tests/src/main.rs"
    click TestSuite "https://github.com/databendlabs/openraft/tree/main/tests/tests/"

    %% Styles
    classDef core-protocol fill:#E8F4FD,stroke:#0288D1,color:#01579B
    classDef interface fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef implementation fill:#ECEFF1,stroke:#546E7A,color:#263238
    classDef examples fill:#FFF3E0,stroke:#FB8C00,color:#E65100
    classDef config fill:#F3E5F5,stroke:#8E24AA,color:#4A148C
```
