# delta-io/delta-kernel-rs architecture

> Architecture diagram of the GitHub repository delta-io/delta-kernel-rs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/delta-io/delta-kernel-rs
- Repository: https://github.com/delta-io/delta-kernel-rs

## Overview

##### Main Components

a. Core Components:
- Delta Kernel Core (central component)
- Engine Trait Interface
- Default Engine Implementation
- Sync Engine Implementation
- FFI Layer

b. Supporting Components:
- Storage Layer
- Table Operations Layer
- Transaction Management
- Schema Management

##### Component Relationships

**Top Level (Consumer APIs):**
**[FFI Interface] -----> [Kernel Public API]**

**Middle Layer:         [Core Kernel]**
- /      |       \
**Lower Layer:  [Engine] [Schema] [Transaction]**

**Bottom Layer: [Storage]**

##### Key Elements to Highlight

a. Data Flow:
- How data flows from storage through the Engine trait
- Demonstrate the transaction flow for write operations

b. Interface Boundaries:
- Clear separation between the core kernel and engine implementations
- FFI boundary for C/C++ interoperability

##### Implementation Details

- Trait definitions (interfaces)
- Concrete implementations
- Optional components (feature-flagged)
- External dependencies

**[Table API] --> [Snapshot Management] --> [Version Control]**

- v               v                         v
**[Scan Operations] [Schema Validation] [Transaction Management]**

**[Engine Trait]**

- > [Default Engine (Async)]

- > [Tokio Runtime]

- > [Sync Engine]
- > [Local Storage Only]

##### Technical Details

- Storage abstraction interfaces
- Transaction isolation boundaries
- Schema evolution paths
- Data format conversions

- The modular nature of the engine trait
- Clear separation of concerns
- Optional feature integration points
- Async boundary isolation
- FFI interface points

Remember to maintain a balance between showing enough detail to understand the system while keeping it clear and readable. Use a hierarchical layout to show dependencies and data flow, with the core kernel at the center and supporting components arranged around it.

This design should help developers understand both the high-level architecture and the specific implementation details of the Delta Kernel project.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "FFI Layer"
        FFI["FFI Interface"]:::ffi
    end

    subgraph "Public API Layer"
        KernelAPI["Kernel Public API"]:::core
    end

    subgraph "Core Layer"
        CoreKernel["Delta Kernel Core"]:::core
        TableOps["Table Operations"]:::core
        SchemaOps["Schema Management"]:::core
        TransactionMgr["Transaction Management"]:::core
        SnapshotMgr["Snapshot Management"]:::core
        TableChanges["Table Changes"]:::core
        Actions["Actions Handler"]:::core
        ExprEngine["Expression Engine"]:::core
        ScanOps["Scan Operations"]:::core
    end

    subgraph "Engine Layer"
        EngineTrait["Engine Trait Interface"]:::engine
        DefaultEngine["Default Engine Implementation"]:::engine
        SyncEngine["Sync Engine Implementation"]:::engine
    end

    subgraph "Integration Layer"
        ArrowIntegration["Arrow Integration"]:::integration
        Storage["Storage Layer"]:::storage
    end

    %% Connections
    FFI --> KernelAPI
    KernelAPI --> CoreKernel

    CoreKernel --> TableOps
    CoreKernel --> SchemaOps
    CoreKernel --> TransactionMgr
    CoreKernel --> SnapshotMgr
    CoreKernel --> TableChanges
    CoreKernel --> Actions
    CoreKernel --> ExprEngine
    CoreKernel --> ScanOps

    CoreKernel --> EngineTrait
    EngineTrait --> DefaultEngine
    EngineTrait --> SyncEngine

    DefaultEngine --> ArrowIntegration
    DefaultEngine --> Storage
    SyncEngine --> Storage

    %% Click Events
    click CoreKernel "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/lib.rs"
    click EngineTrait "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/engine/mod.rs"
    click DefaultEngine "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/engine/default/mod.rs"
    click SyncEngine "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/engine/sync/mod.rs"
    click FFI "https://github.com/delta-io/delta-kernel-rs/blob/main/ffi/src/lib.rs"
    click ArrowIntegration "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/engine/arrow_conversion.rs"
    click Storage "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/engine/default/storage.rs"
    click TableOps "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/table.rs"
    click TransactionMgr "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/transaction.rs"
    click SchemaOps "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/schema.rs"
    click ExprEngine "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/expressions/mod.rs"
    click ScanOps "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/scan/mod.rs"
    click Actions "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/actions/mod.rs"
    click TableChanges "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/table_changes/mod.rs"
    click SnapshotMgr "https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/snapshot.rs"

    %% Styles
    classDef core fill:#f9f,stroke:#333,stroke-width:2px
    classDef engine fill:#bbf,stroke:#333,stroke-width:2px
    classDef integration fill:#bfb,stroke:#333,stroke-width:2px
    classDef storage fill:#fb9,stroke:#333,stroke-width:2px
    classDef ffi fill:#ff9,stroke:#333,stroke-width:2px
```
