# apache/gluten architecture

> Architecture diagram of the GitHub repository apache/gluten, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/gluten
- Repository: https://github.com/apache/gluten

## Overview

Gluten is a Spark SQL middle layer that delegates eligible physical-plan work to native engines while retaining Spark’s distributed execution and fallback control flow. The main workflow converts Spark plans for backend execution, exchanges columnar batches across the JVM/native boundary, and returns results through Spark’s columnar APIs. The tree shows Bolt and ClickHouse integration, with memory managers for Bolt and Velox; README-documented plan conversion, fallback, metrics, and shuffle capabilities are represented where supported. Excerpts are partial, so uncertain call wiring is omitted.

Main entry points: [`cpp-ch/local-engine/local_engine_jni.cpp`](https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/local_engine_jni.cpp) (ClickHouse Engine), [`backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala`](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala) (Columnar Operators), [`backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltTransformerApi.scala`](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltTransformerApi.scala) (Plan Conversion API), [`backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala`](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala) (Bolt Backend)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_plan["Spark Integration"]
  node_bolt_backend["Bolt Backend<br/>[BoltBackend.scala]"]
  node_ch_backend["ClickHouse Backend<br/>[CHBackend.scala]"]
  node_plan_api["Plan Conversion API"]
  node_fallback["Fallback Transforms"]
  node_iceberg["Iceberg Writes"]
end

subgraph group_runtime["Native Execution"]
  node_bolt_engine["Bolt Engine"]
  node_ch_engine["ClickHouse Engine"]
  node_operators["Columnar Operators"]
  node_ch_join["Broadcast Join Build"]
  node_kafka["Kafka Source"]
  node_merge_tree["MergeTree I/O"]
end

subgraph group_exchange["Data Exchange"]
  node_jni_batch["Batch JNI"]
  node_shuffle["Columnar Shuffle"]
  node_jni_executor["Native Executor JNI"]
  node_result["Columnar Results"]
end

subgraph group_services["Runtime Services"]
  node_memory["Native Memory"]
  node_metrics["Execution Metrics"]
end

node_spark(("Spark SQL"))

node_spark -->|"select backend"| node_bolt_backend
node_spark -->|"select backend"| node_ch_backend
node_bolt_backend -->|"provides conversion"| node_plan_api
node_ch_backend -->|"provides conversion"| node_plan_api
node_plan_api -->|"transform plans"| node_operators
node_operators -->|"submit execution"| node_jni_executor
node_jni_executor -->|"create executor"| node_ch_engine
node_jni_batch -->|"exchange batches"| node_ch_engine
node_ch_engine -->|"produce batches"| node_result
node_result -->|"return columnar data"| node_spark
node_spark -.->|"unsupported operators"| node_fallback
node_spark -->|"shuffle batches"| node_shuffle
node_shuffle -->|"transfer shuffle data"| node_jni_executor
node_ch_join -->|"build join table"| node_ch_engine
node_ch_engine -.->|"read source"| node_kafka
node_ch_engine -->|"read or write parts"| node_merge_tree
node_spark -.->|"execute table writes"| node_iceberg
node_bolt_engine -->|"allocate memory"| node_memory
node_operators -->|"report metrics"| node_metrics

click node_bolt_backend "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala"
click node_ch_backend "https://github.com/apache/gluten/blob/main/backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHBackend.scala"
click node_plan_api "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltTransformerApi.scala"
click node_fallback "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/BoltColumnarToRowExec.scala"
click node_bolt_engine "https://github.com/apache/gluten/blob/main/cpp/bolt/memory/BoltMemoryManager.cc"
click node_ch_engine "https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/local_engine_jni.cpp"
click node_operators "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala"
click node_ch_join "https://github.com/apache/gluten/blob/main/backends-clickhouse/src/main/java/org/apache/gluten/vectorized/StorageJoinBuilder.java"
click node_jni_batch "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/java/org/apache/gluten/columnarbatch/BoltColumnarBatchJniWrapper.java"
click node_shuffle "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala"
click node_jni_executor "https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/local_engine_jni.cpp"
click node_memory "https://github.com/apache/gluten/blob/main/cpp/bolt/memory/BoltMemoryManager.cc"
click node_metrics "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala"
click node_kafka "https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/Storages/Kafka/ReadFromGlutenStorageKafka.cpp"
click node_merge_tree "https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/Storages/MergeTree/SparkStorageMergeTree.cpp"
click node_iceberg "https://github.com/apache/gluten/blob/main/backends-bolt/src-iceberg/main/scala/org/apache/gluten/execution/BoltIcebergWriteToDataSourceV2Exec.scala"
click node_result "https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/ArrowColumnarToBoltColumnarExec.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bolt_backend,node_ch_backend,node_plan_api,node_fallback,node_iceberg toneBlue
class node_bolt_engine,node_ch_engine,node_operators,node_ch_join,node_kafka,node_merge_tree toneAmber
class node_jni_batch,node_shuffle,node_jni_executor,node_result toneMint
class node_memory,node_metrics toneRose
class node_spark toneIndigo
```

## Components

### Spark Integration

- [Bolt Backend](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala): `backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala`
- [ClickHouse Backend](https://github.com/apache/gluten/blob/main/backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHBackend.scala): `backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHBackend.scala`
- [Plan Conversion API](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltTransformerApi.scala): `backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltTransformerApi.scala`
- [Fallback Transforms](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/BoltColumnarToRowExec.scala): `backends-bolt/src/main/scala/org/apache/gluten/execution/BoltColumnarToRowExec.scala`
- [Iceberg Writes](https://github.com/apache/gluten/blob/main/backends-bolt/src-iceberg/main/scala/org/apache/gluten/execution/BoltIcebergWriteToDataSourceV2Exec.scala): `backends-bolt/src-iceberg/main/scala/org/apache/gluten/execution/BoltIcebergWriteToDataSourceV2Exec.scala`

### Native Execution

- [Bolt Engine](https://github.com/apache/gluten/blob/main/cpp/bolt/memory/BoltMemoryManager.cc): `cpp/bolt/memory/BoltMemoryManager.cc`
- [ClickHouse Engine](https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/local_engine_jni.cpp): `cpp-ch/local-engine/local_engine_jni.cpp`
- [Columnar Operators](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala): `backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala`
- [Broadcast Join Build](https://github.com/apache/gluten/blob/main/backends-clickhouse/src/main/java/org/apache/gluten/vectorized/StorageJoinBuilder.java): `backends-clickhouse/src/main/java/org/apache/gluten/vectorized/StorageJoinBuilder.java`
- [Kafka Source](https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/Storages/Kafka/ReadFromGlutenStorageKafka.cpp): `cpp-ch/local-engine/Storages/Kafka/ReadFromGlutenStorageKafka.cpp`
- [MergeTree I/O](https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/Storages/MergeTree/SparkStorageMergeTree.cpp): `cpp-ch/local-engine/Storages/MergeTree/SparkStorageMergeTree.cpp`

### Data Exchange

- [Batch JNI](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/java/org/apache/gluten/columnarbatch/BoltColumnarBatchJniWrapper.java): `backends-bolt/src/main/java/org/apache/gluten/columnarbatch/BoltColumnarBatchJniWrapper.java`
- [Columnar Shuffle](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala): `backends-bolt/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala`
- [Native Executor JNI](https://github.com/apache/gluten/blob/main/cpp-ch/local-engine/local_engine_jni.cpp): `cpp-ch/local-engine/local_engine_jni.cpp`
- [Columnar Results](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/execution/ArrowColumnarToBoltColumnarExec.scala): `backends-bolt/src/main/scala/org/apache/gluten/execution/ArrowColumnarToBoltColumnarExec.scala`

### Runtime Services

- [Native Memory](https://github.com/apache/gluten/blob/main/cpp/bolt/memory/BoltMemoryManager.cc): `cpp/bolt/memory/BoltMemoryManager.cc`
- [Execution Metrics](https://github.com/apache/gluten/blob/main/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala): `backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala`

### Other components

- **Spark SQL**

## Connections

- Spark SQL → Bolt Backend: select backend
- Spark SQL → ClickHouse Backend: select backend
- Bolt Backend → Plan Conversion API: provides conversion
- ClickHouse Backend → Plan Conversion API: provides conversion
- Plan Conversion API → Columnar Operators: transform plans
- Columnar Operators → Native Executor JNI: submit execution
- Native Executor JNI → ClickHouse Engine: create executor
- Batch JNI → ClickHouse Engine: exchange batches
- ClickHouse Engine → Columnar Results: produce batches
- Columnar Results → Spark SQL: return columnar data
- Spark SQL → Fallback Transforms: unsupported operators
- Spark SQL → Columnar Shuffle: shuffle batches
- Columnar Shuffle → Native Executor JNI: transfer shuffle data
- Broadcast Join Build → ClickHouse Engine: build join table
- ClickHouse Engine → Kafka Source: read source
- ClickHouse Engine → MergeTree I/O: read or write parts
- Spark SQL → Iceberg Writes: execute table writes
- Bolt Engine → Native Memory: allocate memory
- Columnar Operators → Execution Metrics: report metrics