How goeddie/spark-connect-dotnet works
Main Components
a) Client-Side Components (Top Level):
• .NET Application layer using a rectangle
• Main Spark.Connect.Dotnet client library components:
• SparkSession (entry point)
• DataFrame API
• SQL Functions
• Type System
b) Communication Layer (Middle):
• gRPC Communication Channel
• Protocol Buffers (show the proto files as a component)
• Request/Response Flow
c) Server-Side Components (Bottom):
• Apache Spark Connect Server
• Spark SQL Engine
• Spark Execution Engine
Key Relationships to Highlight
• Bidirectional communication between .NET client and Spark Server
• Protocol buffer serialization/deserialization points
• Async operation flows
b) Component Dependencies:
• SparkSession builder to the gRPC client
• Link DataFrame operations to protocol buffer messages
• Type system mappings between .NET and Spark
• .NET Application |
• Spark.Connect | |
• DataFrame API | |
• SQL Functions | |
• Type System | |
• ↕
• gRPC Layer |
• Protocol Buffers |
• ↕
• Spark Connect Server |
• Spark SQL Engine |
• Execution Engine |
Important Details
• Version compatibility notes
• Error handling flow
• Authentication/security layer (especially for Databricks)
• Data type conversion paths
• Session management
Implementation Notes
• Public APIs vs internal components
• Where custom extensions can be added
• Test coverage boundaries
• Configuration points
Integration points with different Spark environments (standalone, Databricks)
This design should help developers understand both the high-level architecture and the specific implementation details that make this library unique in the Spark ecosystem.