How apache/gluten works
Gluten is a Spark SQL middle layer that delegates eligible physical-plan work to native engines while retaining Spark’s distributed execution and fallback control flow. The main workflow converts Spark plans for backend execution, exchanges columnar batches across the JVM/native boundary, and returns results through Spark’s columnar APIs. The tree shows Bolt and ClickHouse integration, with memory managers for Bolt and Velox; README-documented plan conversion, fallback, metrics, and shuffle capabilities are represented where supported. Excerpts are partial, so uncertain call wiring is omitted.
Main entry points: local_engine_jni.cpp ClickHouse Engine · HashAggregateExecTransformer.scala Columnar Operators · BoltTransformerApi.scala Plan Conversion API · BoltBackend.scala Bolt Backend
Components 18
Spark Integration
- Bolt Backend
backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala - ClickHouse Backend
backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHBackend.scala - Plan Conversion API
backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltTransformerApi.scala - Fallback Transforms
backends-bolt/src/main/scala/org/apache/gluten/execution/BoltColumnarToRowExec.scala - Iceberg Writes
backends-bolt/src-iceberg/main/scala/org/apache/gluten/execution/BoltIcebergWriteToDataSourceV2Exec.scala
Native Execution
- Bolt Engine
cpp/bolt/memory/BoltMemoryManager.cc - ClickHouse Engine
cpp-ch/local-engine/local_engine_jni.cpp - Columnar Operators
backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala - Broadcast Join Build
backends-clickhouse/src/main/java/org/apache/gluten/vectorized/StorageJoinBuilder.java - Kafka Source
cpp-ch/local-engine/Storages/Kafka/ReadFromGlutenStorageKafka.cpp - MergeTree I/O
cpp-ch/local-engine/Storages/MergeTree/SparkStorageMergeTree.cpp
Data Exchange
- Batch JNI
backends-bolt/src/main/java/org/apache/gluten/columnarbatch/BoltColumnarBatchJniWrapper.java - Columnar Shuffle
backends-bolt/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala - Native Executor JNI
cpp-ch/local-engine/local_engine_jni.cpp - Columnar Results
backends-bolt/src/main/scala/org/apache/gluten/execution/ArrowColumnarToBoltColumnarExec.scala
Runtime Services
- Native Memory
cpp/bolt/memory/BoltMemoryManager.cc - Execution Metrics
backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala
Other components
- Spark SQL
Connections 19
- Spark SQL → Bolt Backend: select backend
- Spark SQL → ClickHouse Backend: select backend
- Bolt Backend → Plan Conversion API: provides conversion
- ClickHouse Backend → Plan Conversion API: provides conversion
- Plan Conversion API → Columnar Operators: transform plans
- Columnar Operators → Native Executor JNI: submit execution
- Native Executor JNI → ClickHouse Engine: create executor
- Batch JNI → ClickHouse Engine: exchange batches
- ClickHouse Engine → Columnar Results: produce batches
- Columnar Results → Spark SQL: return columnar data
- Spark SQL → Fallback Transforms: unsupported operators
- Spark SQL → Columnar Shuffle: shuffle batches
- Columnar Shuffle → Native Executor JNI: transfer shuffle data
- Broadcast Join Build → ClickHouse Engine: build join table
- ClickHouse Engine → Kafka Source: read source
- ClickHouse Engine → MergeTree I/O: read or write parts
- Spark SQL → Iceberg Writes: execute table writes
- Bolt Engine → Native Memory: allocate memory
- Columnar Operators → Execution Metrics: report metrics