# delta-io/delta-rs architecture

> Architecture diagram of the GitHub repository delta-io/delta-rs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/delta-io/delta-rs
- Repository: https://github.com/delta-io/delta-rs

## Overview

##### Main Components

a. Core Components:
- Delta Table Management System
- Storage Layer Interface
- Protocol Implementation
- Transaction Management
- Schema Management
- Operations Engine

b. Integration Components:
- Cloud Storage Adapters (AWS S3, Azure, GCS, HDFS)
- Language Bindings (Python)

##### Key Architectural Patterns

a. Layered Architecture:
**┌─────────────────────────────────────────────┐**
- Client Interfaces              │
- (Python API, Rust Native API)        │
- ┤
- Operations Layer                  │
- (Create, Read, Write, Merge, Optimize)     │
- ┤
- Transaction Layer                 │
- (ACID Guarantees, Concurrency)         │
- ┤
- Storage Layer                     │
- (Cloud Storage, Local FS Adapters)      │
- ┘

##### Detailed Component Breakdown

a. Core Engine:
**┌─────────────────────────────────────────────────────┐**
- Delta Core Engine                  │
- ┬──────────────────┬──────────────────┤
- Protocol   │   Transaction    │     Schema        │
- Handler    │   Manager        │     Manager       │
- ┼──────────────────┼──────────────────┤
- Log Store  │   State Manager  │  Data Validator   │
- ┴──────────────────┴──────────────────┘

b. Storage Integration:
**┌─────────────────────────────────────────────┐**
- Storage Interface                │
- ┬───────────┬──────────┬─────────┤
- AWS     │  Azure    │   GCS    │  HDFS   │
- S3      │  Storage  │          │         │
- ┴───────────┴──────────┴─────────┘

##### Data Flow Representation

- Read/Write operations flow
- Transaction coordination
- Storage interaction patterns
- Integration points with external systems

##### Key Elements to Highlight

a. Transaction Management:
- Optimistic concurrency control
- ACID compliance mechanisms
- Checkpoint management

b. Data Organization:
- Parquet file management
- Delta log structure
- Partition handling

##### Integration Points

Connections to:
- Python bindings
- External storage systems
- Other processing frameworks

**🔷 Core Component**
**🔶 Storage Adapter**
**🔺 Integration Point**
**↔️ Data Flow**
**⚡ Transaction Boundary**

##### Storage abstraction

Remember to maintain a balance between detail and clarity, focusing on the key architectural components while avoiding overwhelming detail at the implementation level.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style Definitions
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef storage fill:#2ecc71,stroke:#27ae60,color:white
    classDef integration fill:#e67e22,stroke:#d35400,color:white
    classDef transaction fill:#f1c40f,stroke:#f39c12,color:black
    classDef infrastructure fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Client Interface Layer
    subgraph ClientLayer
        PythonAPI["Python API"]:::integration
        RustAPI["Rust Native API"]:::integration
    end

    %% Operations Layer
    subgraph OperationsLayer
        OpEngine["Operations Engine"]:::core
        TransactionMgr["Transaction Manager"]:::transaction
        SchemaManager["Schema Manager"]:::core
    end

    %% Core Engine Layer
    subgraph CoreLayer
        DeltaCore["Delta Core Engine"]:::core
        ProtocolHandler["Protocol Handler"]:::core
        StateManager["State Manager"]:::core
        LogStore["Log Store"]:::core
        DataValidator["Data Validator"]:::core
    end

    %% Storage Layer
    subgraph StorageLayer
        StorageInterface["Storage Interface"]:::storage
        S3["AWS S3"]:::storage
        Azure["Azure Storage"]:::storage
        GCS["Google Cloud Storage"]:::storage
        HDFS["HDFS"]:::storage
    end

    %% Integration Components
    subgraph IntegrationLayer
        ArrowIntegration["Arrow Integration"]:::integration
        DataFusionIntegration["DataFusion Integration"]:::integration
    end

    %% Relationships
    PythonAPI --> OpEngine
    RustAPI --> OpEngine
    OpEngine --> TransactionMgr
    OpEngine --> SchemaManager
    TransactionMgr --> DeltaCore
    SchemaManager --> DeltaCore
    DeltaCore --> ProtocolHandler
    DeltaCore --> StateManager
    DeltaCore --> LogStore
    DeltaCore --> DataValidator
    DeltaCore --> StorageInterface
    StorageInterface --> S3
    StorageInterface --> Azure
    StorageInterface --> GCS
    StorageInterface --> HDFS
    DeltaCore --> ArrowIntegration
    DeltaCore --> DataFusionIntegration

    %% Click Events
    click OpEngine "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/operations/"
    click TransactionMgr "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/operations/transaction/"
    click SchemaManager "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/schema/"
    click ProtocolHandler "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/protocol/"
    click StorageInterface "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/storage/"
    click S3 "https://github.com/delta-io/delta-rs/tree/main/crates/aws/"
    click Azure "https://github.com/delta-io/delta-rs/tree/main/crates/azure/"
    click GCS "https://github.com/delta-io/delta-rs/tree/main/crates/gcp/"
    click HDFS "https://github.com/delta-io/delta-rs/tree/main/crates/hdfs/"
    click PythonAPI "https://github.com/delta-io/delta-rs/tree/main/python/"
    click ArrowIntegration "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/kernel/arrow/"
    click DataFusionIntegration "https://github.com/delta-io/delta-rs/tree/main/crates/core/src/delta_datafusion/"

    %% Legend
    subgraph Legend
        L1["Core Component"]:::core
        L2["Storage Component"]:::storage
        L3["Integration Point"]:::integration
        L4["Transaction Component"]:::transaction
        L5["Infrastructure"]:::infrastructure
    end
```
