# goeddie/spark-connect-dotnet architecture

> Architecture diagram of the GitHub repository goeddie/spark-connect-dotnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/goeddie/spark-connect-dotnet
- Repository: https://github.com/goeddie/spark-connect-dotnet

## Overview

##### Main Components

a) Client-Side Components (Top Level):
- .NET Application layer using a rectangle
- Main Spark.Connect.Dotnet client library components:
- SparkSession (entry point)
- DataFrame API
- SQL Functions
- Type System

b) Communication Layer (Middle):
- gRPC Communication Channel
- Protocol Buffers (show the proto files as a component)
- Request/Response Flow

c) Server-Side Components (Bottom):
- Apache Spark Connect Server
- Spark SQL Engine
- Spark Execution Engine

##### Key Relationships to Highlight

- Bidirectional communication between .NET client and Spark Server
- Protocol buffer serialization/deserialization points
- Async operation flows

b) Component Dependencies:
- SparkSession builder to the gRPC client
- Link DataFrame operations to protocol buffer messages
- Type system mappings between .NET and Spark

- .NET Application     |

- Spark.Connect   | |
- DataFrame API | |
- SQL Functions | |
- Type System   | |

- ↕

- gRPC Layer          |
- Protocol Buffers      |

- ↕

- Spark Connect Server  |
- Spark SQL Engine   |
- Execution Engine   |

##### Important Details

- Version compatibility notes
- Error handling flow
- Authentication/security layer (especially for Databricks)
- Data type conversion paths
- Session management

##### Implementation Notes

- Public APIs vs internal components
- Where custom extensions can be added
- Test coverage boundaries
- Configuration points

##### Integration points with different Spark environments (standalone, Databricks)

This design should help developers understand both the high-level architecture and the specific implementation details that make this library unique in the Spark ecosystem.

## Diagram (Mermaid)

```mermaid
%%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '16px'}}}%%
flowchart TB
    subgraph Client [".NET Application Layer"]
        direction TB
        App[".NET Application"]
        subgraph SparkConnectDotnet ["Spark.Connect.Dotnet Library"]
            direction TB
            Session["SparkSession"]
            DataFrame["DataFrame API"]
            SQL["SQL Functions"]
            TypeSystem["Type System"]
            Reader["DataFrame Reader"]
            Writer["DataFrame Writer"]
            Streaming["Streaming Components"]
            Exceptions["Exception Handling"]
            Databricks["Databricks Integration"]
        end
        App --> SparkConnectDotnet
    end

    subgraph Communication ["Communication Layer"]
        direction TB
        GRPC["gRPC Channel"]
        Proto["Protocol Buffers"]
    end

    subgraph Server ["Apache Spark Server Layer"]
        direction TB
        SparkConnect["Spark Connect Server"]
        SQLEngine["Spark SQL Engine"]
        ExecEngine["Spark Execution Engine"]
    end

    %% Connections
    Session --> GRPC
    DataFrame --> GRPC
    SQL --> GRPC
    GRPC <--> Proto
    Proto <--> SparkConnect
    SparkConnect --> SQLEngine
    SQLEngine --> ExecEngine

    %% Styling
    classDef client fill:#a8d1f0,stroke:#333,stroke-width:2px
    classDef communication fill:#90EE90,stroke:#333,stroke-width:2px
    classDef server fill:#FFB347,stroke:#333,stroke-width:2px
    classDef component fill:#f9f9f9,stroke:#333,stroke-width:1px

    class App,Session,DataFrame,SQL,TypeSystem,Reader,Writer,Streaming,Exceptions,Databricks client
    class GRPC,Proto communication
    class SparkConnect,SQLEngine,ExecEngine server

    %% Click events
    click Session "https://github.com/GoEddie/spark-connect-dotnet/blob/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/SparkSession.cs"
    click DataFrame "https://github.com/GoEddie/spark-connect-dotnet/blob/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/DataFrame.cs"
    click SQL "https://github.com/GoEddie/spark-connect-dotnet/blob/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/Functions.cs"
    click TypeSystem "https://github.com/GoEddie/spark-connect-dotnet/tree/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/Types/"
    click Proto "https://github.com/GoEddie/spark-connect-dotnet/tree/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet.GrpcClient/spark/connect/"
    click GRPC "https://github.com/GoEddie/spark-connect-dotnet/tree/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Grpc/"
    click Exceptions "https://github.com/GoEddie/spark-connect-dotnet/tree/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Grpc/SparkExceptions/"
    click Databricks "https://github.com/GoEddie/spark-connect-dotnet/tree/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Databricks/"
    click Streaming "https://github.com/GoEddie/spark-connect-dotnet/tree/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/Streaming/"
    click Reader "https://github.com/GoEddie/spark-connect-dotnet/blob/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/DataFrameReader.cs"
    click Writer "https://github.com/GoEddie/spark-connect-dotnet/blob/main/src/Spark.Connect.Dotnet/Spark.Connect.Dotnet/Sql/DataFrameWriter.cs"

    %% Legend
    subgraph Legend
        direction TB
        Client1[".NET Client Components"]:::client
        Comm1["Communication Layer"]:::communication
        Server1["Spark Server Components"]:::server
    end
```
