# apache/datafusion architecture

> Architecture diagram of the GitHub repository apache/datafusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/datafusion
- Repository: https://github.com/apache/datafusion
- Video tour (about a minute): https://gitdiagram.com/apache/datafusion/video

## Overview

Apache DataFusion is an extensible Rust query engine using Arrow’s columnar format. Users submit SQL or DataFrame queries; the engine plans and optimizes them, creates a physical plan, reads supported data sources, and executes a streaming, parallel query. The graph also preserves major extension and runtime capabilities, including catalogs, functions, caching, spilling, the CLI, and FFI. Some internal links are documented only at the product level; those are cited to the README rather than inferred from unsampled code.

Main entry points: [`datafusion/expr/src/logical_plan/plan.rs`](https://github.com/apache/datafusion/blob/main/datafusion/expr/src/logical_plan/plan.rs) (Logical plans), [`datafusion/physical-plan/src/stream.rs`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/stream.rs) (Query execution), [`datafusion/optimizer/src/optimizer.rs`](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/optimizer.rs) (Logical optimizer), [`datafusion/core/src/physical_planner.rs`](https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_planner.rs) (Physical planner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_sql_api["SQL API<br/>[query.rs]"]
  node_dataframe_api["DataFrame API<br/>[mod.rs]"]
  node_cli["Command-line interface<br/>[main.rs]"]
  node_object_storage["CLI object storage<br/>[object_storage.rs]"]
  node_cli_context["CLI context<br/>[cli_context.rs]"]
end

subgraph group_planning["Query planning"]
  node_sql_planner["SQL planner<br/>[planner.rs]"]
  node_logical_plan["Logical plans<br/>[plan.rs]"]
  node_optimizer["Logical optimizer<br/>[optimizer.rs]"]
  node_predicate_rewrite["Subquery rewrites"]
  node_scalar_rewrite["Scalar rewrites"]
  node_physical_planner["Physical planner"]
end

subgraph group_runtime["Execution runtime"]
  node_execution["Query execution<br/>[stream.rs]"]
  node_scalar_execution["Scalar subqueries<br/>[scalar_subquery.rs]"]
  node_cache["Metadata caches<br/>[cache_manager.rs]"]
  node_disk_manager["Spill files<br/>[disk_manager.rs]"]
  node_spill_manager["Spill management<br/>[spill_manager.rs]"]
end

subgraph group_data["Data and catalogs"]
  node_datasources["Data sources<br/>[provider.rs]"]
  node_file_formats["File formats<br/>[file_format.rs]"]
  node_catalog["Catalogs<br/>[catalog.rs]"]
end

subgraph group_extensions["Functions and extensions"]
  node_functions["Built-in functions<br/>[average.rs]"]
  node_ffi["Foreign interface<br/>[query_planner.rs]"]
  node_substrait["Substrait exchange<br/>[subquery.rs]"]
end

node_user(("Query user"))

node_user -->|"submits SQL"| node_sql_api
node_user -->|"builds queries"| node_dataframe_api
node_user -->|"runs queries"| node_cli
node_sql_api -->|"plans SQL"| node_sql_planner
node_dataframe_api -.->|"builds plans"| node_logical_plan
node_sql_planner -->|"produces plans"| node_logical_plan
node_logical_plan -->|"optimizes plans"| node_optimizer
node_optimizer -->|"dispatches rules"| node_predicate_rewrite
node_optimizer -->|"dispatches rules"| node_scalar_rewrite
node_optimizer -->|"plans execution"| node_physical_planner
node_physical_planner -->|"creates plans"| node_execution
node_execution -->|"reads data"| node_datasources
node_datasources -->|"reads formats"| node_file_formats
node_execution -.->|"executes subqueries"| node_scalar_execution
node_spill_manager -->|"creates spill files"| node_disk_manager
node_execution -.->|"spills batches"| node_spill_manager
node_cache -.->|"caches metadata"| node_execution
node_logical_plan -.->|"resolves tables"| node_catalog
node_logical_plan -.->|"uses expressions"| node_functions
node_ffi -->|"delegates planning"| node_physical_planner
node_substrait -.->|"imports plans"| node_logical_plan
node_cli -.->|"uses context"| node_cli_context
node_cli_context -->|"configures storage"| node_object_storage

click node_sql_api "https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs"
click node_dataframe_api "https://github.com/apache/datafusion/blob/main/datafusion/core/src/dataframe/mod.rs"
click node_cli "https://github.com/apache/datafusion/blob/main/datafusion-cli/src/main.rs"
click node_sql_planner "https://github.com/apache/datafusion/blob/main/datafusion/sql/src/planner.rs"
click node_logical_plan "https://github.com/apache/datafusion/blob/main/datafusion/expr/src/logical_plan/plan.rs"
click node_optimizer "https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/optimizer.rs"
click node_predicate_rewrite "https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/decorrelate_predicate_subquery.rs"
click node_scalar_rewrite "https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/scalar_subquery_to_join.rs"
click node_physical_planner "https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_planner.rs"
click node_execution "https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/stream.rs"
click node_scalar_execution "https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/scalar_subquery.rs"
click node_cache "https://github.com/apache/datafusion/blob/main/datafusion/execution/src/cache/cache_manager.rs"
click node_disk_manager "https://github.com/apache/datafusion/blob/main/datafusion/execution/src/disk_manager.rs"
click node_spill_manager "https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/spill/spill_manager.rs"
click node_datasources "https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/provider.rs"
click node_file_formats "https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/file_format.rs"
click node_catalog "https://github.com/apache/datafusion/blob/main/datafusion/catalog/src/catalog.rs"
click node_functions "https://github.com/apache/datafusion/blob/main/datafusion/functions-aggregate/src/average.rs"
click node_ffi "https://github.com/apache/datafusion/blob/main/datafusion/ffi/src/query_planner.rs"
click node_substrait "https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/consumer/expr/subquery.rs"
click node_object_storage "https://github.com/apache/datafusion/blob/main/datafusion-cli/src/object_storage.rs"
click node_cli_context "https://github.com/apache/datafusion/blob/main/datafusion-cli/src/cli_context.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sql_api,node_dataframe_api,node_cli,node_object_storage,node_cli_context,node_user toneBlue
class node_sql_planner,node_logical_plan,node_optimizer,node_predicate_rewrite,node_scalar_rewrite,node_physical_planner toneAmber
class node_execution,node_scalar_execution,node_cache,node_disk_manager,node_spill_manager toneMint
class node_datasources,node_file_formats,node_catalog toneRose
class node_functions,node_ffi,node_substrait toneIndigo
```

## Components

### User interfaces

- [SQL API](https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs): `datafusion/sql/src/query.rs`
- [DataFrame API](https://github.com/apache/datafusion/blob/main/datafusion/core/src/dataframe/mod.rs): `datafusion/core/src/dataframe/mod.rs`
- [Command-line interface](https://github.com/apache/datafusion/blob/main/datafusion-cli/src/main.rs): `datafusion-cli/src/main.rs`
- [CLI object storage](https://github.com/apache/datafusion/blob/main/datafusion-cli/src/object_storage.rs): `datafusion-cli/src/object_storage.rs`
- [CLI context](https://github.com/apache/datafusion/blob/main/datafusion-cli/src/cli_context.rs): `datafusion-cli/src/cli_context.rs`

### Query planning

- [SQL planner](https://github.com/apache/datafusion/blob/main/datafusion/sql/src/planner.rs): `datafusion/sql/src/planner.rs`
- [Logical plans](https://github.com/apache/datafusion/blob/main/datafusion/expr/src/logical_plan/plan.rs): `datafusion/expr/src/logical_plan/plan.rs`
- [Logical optimizer](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/optimizer.rs): `datafusion/optimizer/src/optimizer.rs`
- [Subquery rewrites](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/decorrelate_predicate_subquery.rs): `datafusion/optimizer/src/decorrelate_predicate_subquery.rs`
- [Scalar rewrites](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/scalar_subquery_to_join.rs): `datafusion/optimizer/src/scalar_subquery_to_join.rs`
- [Physical planner](https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_planner.rs): `datafusion/core/src/physical_planner.rs`

### Execution runtime

- [Query execution](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/stream.rs): `datafusion/physical-plan/src/stream.rs`
- [Scalar subqueries](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/scalar_subquery.rs): `datafusion/physical-plan/src/scalar_subquery.rs`
- [Metadata caches](https://github.com/apache/datafusion/blob/main/datafusion/execution/src/cache/cache_manager.rs): `datafusion/execution/src/cache/cache_manager.rs`
- [Spill files](https://github.com/apache/datafusion/blob/main/datafusion/execution/src/disk_manager.rs): `datafusion/execution/src/disk_manager.rs`
- [Spill management](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/spill/spill_manager.rs): `datafusion/physical-plan/src/spill/spill_manager.rs`

### Data and catalogs

- [Data sources](https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/provider.rs): `datafusion/core/src/datasource/provider.rs`
- [File formats](https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/file_format.rs): `datafusion/datasource-parquet/src/file_format.rs`
- [Catalogs](https://github.com/apache/datafusion/blob/main/datafusion/catalog/src/catalog.rs): `datafusion/catalog/src/catalog.rs`

### Functions and extensions

- [Built-in functions](https://github.com/apache/datafusion/blob/main/datafusion/functions-aggregate/src/average.rs): `datafusion/functions-aggregate/src/average.rs`
- [Foreign interface](https://github.com/apache/datafusion/blob/main/datafusion/ffi/src/query_planner.rs): `datafusion/ffi/src/query_planner.rs`
- [Substrait exchange](https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/consumer/expr/subquery.rs): `datafusion/substrait/src/logical_plan/consumer/expr/subquery.rs`

### Other components

- **Query user**

## Connections

- Query user → SQL API: submits SQL (evidence: `README.md`)
- Query user → DataFrame API: builds queries (evidence: `README.md`)
- Query user → Command-line interface: runs queries (evidence: `README.md`)
- SQL API → SQL planner: plans SQL (evidence: `datafusion/sql/src/query.rs`)
- DataFrame API → Logical plans: builds plans
- SQL planner → Logical plans: produces plans (evidence: `README.md`)
- Logical plans → Logical optimizer: optimizes plans (evidence: `README.md`)
- Logical optimizer → Subquery rewrites: dispatches rules (evidence: `datafusion/optimizer/src/optimizer.rs`)
- Logical optimizer → Scalar rewrites: dispatches rules (evidence: `datafusion/optimizer/src/optimizer.rs`)
- Logical optimizer → Physical planner: plans execution (evidence: `README.md`)
- Physical planner → Query execution: creates plans (evidence: `README.md`)
- Query execution → Data sources: reads data (evidence: `README.md`)
- Data sources → File formats: reads formats (evidence: `README.md`)
- Query execution → Scalar subqueries: executes subqueries (evidence: `datafusion/physical-plan/src/scalar_subquery.rs`)
- Spill management → Spill files: creates spill files (evidence: `datafusion/physical-plan/src/spill/spill_manager.rs`)
- Query execution → Spill management: spills batches (evidence: `datafusion/physical-plan/src/spill/spill_manager.rs`)
- Metadata caches → Query execution: caches metadata
- Logical plans → Catalogs: resolves tables
- Logical plans → Built-in functions: uses expressions
- Foreign interface → Physical planner: delegates planning (evidence: `datafusion/ffi/src/query_planner.rs`)
- Substrait exchange → Logical plans: imports plans
- Command-line interface → CLI context: uses context
- CLI context → CLI object storage: configures storage (evidence: `datafusion-cli/src/cli_context.rs`)
