# tikv/raft-engine architecture

> Architecture diagram of the GitHub repository tikv/raft-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tikv/raft-engine
- Repository: https://github.com/tikv/raft-engine

## Overview

##### Main Components

a. Core Components:
- Memtable System
- Log File Management
- Write Pipeline
- Garbage Collection System
- Storage Layer
- Compression Layer (LZ4)

b. Supporting Components:
- Configuration Management
- Error Handling
- Metrics Collection
- File System Interface

##### Key Relationships to Highlight

- Write flow: Client → Log Batch → Write Pipeline → Log File
- Read flow: Client → Memtable → Log File (if needed)
- GC flow: Purge Trigger → Log File Scanner → Compaction Process

##### Special Elements

- Concurrent Write Queue Management
- Compression Pipeline
- File Rotation Logic
- GC Collaboration System

##### Important Details to Capture

- The log-structured nature of the storage
- The shared log stream between Raft Groups
- The separation between logical and physical deletion
- The collaborative GC mechanism
- The atomic write guarantees

##### Component Relationships

Clear boundaries between:
- In-memory components (Memtable)
- On-disk components (Log Files)
- Temporary states (Write Queue)
- Background processes (GC)

##### Configuration Points

Configurable elements:
- Log file rotation thresholds
- Memory limits (when swap feature is enabled)
- Compression settings
- GC parameters

##### Testing/Monitoring Points

Indicators for:
- Metrics collection points
- Failpoint locations (when enabled)
- Benchmark measurement points

## Diagram (Mermaid)

```mermaid
graph TB
    %% Client Interface Layer
    subgraph ClientLayer
        API[Public API Interface]
        LogBatch[Log Batch Interface]:::logbatch
        Control[Control Interface]:::control
    end

    %% Core Processing Layer
    subgraph CoreLayer
        Memtable[Memtable System]:::memtable
        WritePipeline[Write Pipeline]:::pipeline
        Consistency[Consistency Management]:::consistency
        EventSystem[Event Listener System]:::event
    end

    %% Storage Management Layer
    subgraph StorageLayer
        LogFileManager[Log File Management]:::logfile
        FilePipeFormat[File Pipe Format]:::format
        WriteBarrier[Write Barrier]:::barrier
        Compression[Compression Layer]:::compression
    end

    %% Persistence Layer
    subgraph PersistenceLayer
        FSInterface[File System Interface]:::fsinterface
        PlainLog[Plain Log Implementation]:::plainlog
        UnixLog[Unix Log Implementation]:::unixlog
        ObfuscatedStorage[Obfuscated Storage]:::obfuscated
    end

    %% Supporting Systems
    subgraph SupportSystems
        Config[Configuration Management]:::config
        ErrorHandler[Error Handling]:::error
        Metrics[Metrics Collection]:::metrics
        MemoryMgmt[Memory Management]:::memory
        GC[Garbage Collection]:::gc
    end

    %% Relationships
    API --> LogBatch
    LogBatch --> WritePipeline
    WritePipeline --> LogFileManager
    LogFileManager --> FSInterface
    FSInterface --> |Plain|PlainLog
    FSInterface --> |Unix|UnixLog
    FSInterface --> |Obfuscated|ObfuscatedStorage
    
    Memtable --> LogFileManager
    WriteBarrier --> LogFileManager
    LogFileManager --> FilePipeFormat
    
    GC --> LogFileManager
    Config --> WritePipeline
    Config --> GC
    Config --> Memtable
    
    EventSystem --> Metrics
    Consistency --> WritePipeline
    Control --> Config

    %% Click Events
    click Memtable "https://github.com/tikv/raft-engine/blob/master/src/memtable.rs"
    click LogFileManager "https://github.com/tikv/raft-engine/blob/master/src/file_pipe_log/log_file.rs"
    click WritePipeline "https://github.com/tikv/raft-engine/blob/master/src/file_pipe_log/pipe.rs"
    click GC "https://github.com/tikv/raft-engine/blob/master/src/purge.rs"
    click FSInterface "https://github.com/tikv/raft-engine/tree/master/src/env/log_fd/"
    click Config "https://github.com/tikv/raft-engine/blob/master/src/config.rs"
    click ErrorHandler "https://github.com/tikv/raft-engine/blob/master/src/errors.rs"
    click Metrics "https://github.com/tikv/raft-engine/blob/master/src/metrics.rs"
    click LogBatch "https://github.com/tikv/raft-engine/blob/master/src/log_batch.rs"
    click Consistency "https://github.com/tikv/raft-engine/blob/master/src/consistency.rs"
    click EventSystem "https://github.com/tikv/raft-engine/blob/master/src/event_listener.rs"
    click WriteBarrier "https://github.com/tikv/raft-engine/blob/master/src/write_barrier.rs"
    click Control "https://github.com/tikv/raft-engine/blob/master/ctl/src/lib.rs"
    click FilePipeFormat "https://github.com/tikv/raft-engine/blob/master/src/file_pipe_log/format.rs"
    click MemoryMgmt "https://github.com/tikv/raft-engine/blob/master/src/swappy_allocator.rs"
    click PlainLog "https://github.com/tikv/raft-engine/blob/master/src/env/log_fd/plain.rs"
    click UnixLog "https://github.com/tikv/raft-engine/blob/master/src/env/log_fd/unix.rs"
    click ObfuscatedStorage "https://github.com/tikv/raft-engine/blob/master/src/env/obfuscated.rs"

    %% Styles
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px
    classDef memtable fill:#b3e0ff,stroke:#333
    classDef pipeline fill:#98fb98,stroke:#333
    classDef logfile fill:#87ceeb,stroke:#333
    classDef format fill:#dda0dd,stroke:#333
    classDef barrier fill:#f0e68c,stroke:#333
    classDef compression fill:#ffa07a,stroke:#333
    classDef fsinterface fill:#c0c0c0,stroke:#333
    classDef plainlog fill:#d3d3d3,stroke:#333
    classDef unixlog fill:#d3d3d3,stroke:#333
    classDef obfuscated fill:#d3d3d3,stroke:#333
    classDef config fill:#ffb6c1,stroke:#333
    classDef error fill:#ff6b6b,stroke:#333
    classDef metrics fill:#98fb98,stroke:#333
    classDef memory fill:#dda0dd,stroke:#333
    classDef gc fill:#ffd700,stroke:#333
    classDef logbatch fill:#add8e6,stroke:#333
    classDef consistency fill:#f0e68c,stroke:#333
    classDef event fill:#98fb98,stroke:#333
    classDef control fill:#ffb6c1,stroke:#333
```
