How apache/incubator-gluten works
Spark SQL Engine (JVM)
• This represents the Spark SQL frontend along with Catalyst optimizer.
• It produces the Spark physical plan as part of the usual control flow.
• Optionally, add elements like the DataFrame API, Spark’s UI, and the scheduler.
Gluten Plugin and Query Conversion Module
• Positioned as an interceptor inside SparkSQL, this component loads as a Spark plugin.
• This module also implements fallback logic that checks for operators not supported in the native execution engine. When necessary, it converts columnar data to row-based formats (using Columnar-to-Row (C2R) and Row-to-Columnar (R2C) transformations).
JNI Bridge / Shims
• A dedicated layer that bridges between the JVM (Gluten’s Java modules) and the native side.
• In addition, shims support compatibility with multiple Spark versions. (Mark these as “Shim Layer” components.)
Native Execution Engine(s)
• This large block represents one or more specialized native backends (e.g., ClickHouse backend, Velox backend, and potentially others).
• Data processing is done in native code with its own operator pipelines and optimizations.
Unified Memory Management and Data Exchange
• A memory management component that handles off-heap/native memory allocation and subsequently returns ArrowColumnarBatches.
Fallback / Error Handling Path
• Clearly label a fallback path that converts native columnar data back into Spark’s row format when an operator does not have a native implementation.
• This ensures that users see transparent, correct results even when full offloading is not available.
External Tools and Qualification
• For the Metrics module, show how native engine metrics get transmitted back to the driver for display in the Spark UI.
• Finally, include explicit labeling for key patterns: “Plan Conversion”, “JNI Bridge”, “Fallback Mechanism”, “Unified Memory Management”, and “Native Backends”.