How databendlabs/openraft works
Project Type and Purpose
• This is an open-source Rust library/crate (“openraft”) implementing the Raft consensus protocol as an embeddable engine.
• It offers a pluggable, asynchronous, event-driven Raft core (leader election, log replication, snapshotting, dynamic membership) and exposes trait-based extension points for storage, networking, and state-machine persistence.
• Example applications (under examples/) wire the core to in-memory stores, gRPC transport, RocksDB, Opendal snapshots, single-threaded loops, etc.
• A benchmarking harness (in cluster_benchmark/) and end-to-end tests (tests/) validate correctness and performance.
High-Level Components
a. Raft API Layer
• The public Raft<T> struct drives the node’s lifecycle and exposes methods like trigger_elect(), apply(), add_learner(), trigger_snapshot(), ensure_linearizable().
• Configured via a Config object specifying timeouts, feature flags (heartbeat/elect toggles), and tracing levels.
b. Core Engine
• Submodules in openraft/src/core/ and openraft/src/engine/:
• Event loop (“engine_impl”) dispatching Election, Heartbeat, Replication, Command, Snapshot, Vote handlers.
• State machines managing ServerState, ReplicateState, MembershipState.
• Queueing & batching of Raft messages for throughput.
c. Networking Abstraction
• The RaftNetwork trait in openraft/src/network/ defines RPCs: AppendEntries, Vote, InstallSnapshot.
• Two versions (v1, v2) with adapter modules supporting trait upgrades.
• Examples implement this trait over gRPC (examples/raft-kv-memstore-grpc), simple TCP, or in-process channels.
d. Storage Abstraction
• The RaftLogStorage and snapshot traits under openraft/src/storage/.
• Responsibilities: persisting logs, state machine snapshots, on-disk indexing.
• Examples and stores/ show in-memory and RocksDB implementations.
e. State-Machine Hooks
• The RaftStateMachine trait applied to user data upon commit.
• Serialized commands carried in openraft/src/entry/.
f. Supporting Subsystems
• Type configuration (type_config/) to swap Tokio or monoio runtimes, channel/backpressure implementations.
• Tracing instrumentation via tracing crate.
• Metrics collector in openraft/src/metrics/.
• Testing utilities under openraft/src/testing/.
g. Examples & Benchmarks
• Multiple sample apps, each wiring network, storage, and SM to the core.
• cluster_benchmark/ runs high-throughput put workloads.
Component Interactions and Data Flow
1. Application invokes Raft::new(...) with trait implementations for network, storage, and SM.
2. The Raft core starts an async task:
– Election module monitors timeouts or manual triggers → broadcasts RequestVote RPCs via RaftNetwork.
– Leader on election success handles client writes → appends to log storage, replicates via AppendEntries RPCs.
– Followers receive RPCs, persist to storage, reply, and signal commit.
– Commit notification passed to RaftStateMachine to apply entries.
– Snapshot logic offloads old logs into snapshot storage and replicates via InstallSnapshot RPCs.
3. All subsystems emit metrics and tracing spans.
4. Type-config layers manage task spawning, channel backpressure, and runtime selection.
Architectural Patterns & Principles
• Event-driven, non-blocking Raft state machine using Tokio (async/await).
• Strict separation of core (protocol) and pluggable interfaces (network/storage/SM) via Rust traits.
• Module organization by feature area (config, core, engine, network, storage, replication, membership, testing).
• Examples illustrate dependency-inversion: the core depends only on trait interfaces.
Key Technologies & Libraries
• Rust async ecosystem: Tokio (or alternative runtimes via type_config), futures, channels.
• Traits for RaftLogStorage, RaftNetwork, RaftStateMachine.
• tracing for instrumentation, metrics for runtime stats.
• Example transports: gRPC (via tonic), Opendal, RocksDB, in-memory.