How apache/datafusion works
Apache DataFusion is an extensible Rust query engine using Arrow’s columnar format. Users submit SQL or DataFrame queries; the engine plans and optimizes them, creates a physical plan, reads supported data sources, and executes a streaming, parallel query. The graph also preserves major extension and runtime capabilities, including catalogs, functions, caching, spilling, the CLI, and FFI. Some internal links are documented only at the product level; those are cited to the README rather than inferred from unsampled code.
Main entry points: plan.rs Logical plans · stream.rs Query execution · optimizer.rs Logical optimizer · physical_planner.rs Physical planner
Components 23
User interfaces
- SQL API
datafusion/sql/src/query.rs - DataFrame API
datafusion/core/src/dataframe/mod.rs - Command-line interface
datafusion-cli/src/main.rs - CLI object storage
datafusion-cli/src/object_storage.rs - CLI context
datafusion-cli/src/cli_context.rs
Query planning
- SQL planner
datafusion/sql/src/planner.rs - Logical plans
datafusion/expr/src/logical_plan/plan.rs - Logical optimizer
datafusion/optimizer/src/optimizer.rs - Subquery rewrites
datafusion/optimizer/src/decorrelate_predicate_subquery.rs - Scalar rewrites
datafusion/optimizer/src/scalar_subquery_to_join.rs - Physical planner
datafusion/core/src/physical_planner.rs
Execution runtime
- Query execution
datafusion/physical-plan/src/stream.rs - Scalar subqueries
datafusion/physical-plan/src/scalar_subquery.rs - Metadata caches
datafusion/execution/src/cache/cache_manager.rs - Spill files
datafusion/execution/src/disk_manager.rs - Spill management
datafusion/physical-plan/src/spill/spill_manager.rs
Data and catalogs
- Data sources
datafusion/core/src/datasource/provider.rs - File formats
datafusion/datasource-parquet/src/file_format.rs - Catalogs
datafusion/catalog/src/catalog.rs
Functions and extensions
- Built-in functions
datafusion/functions-aggregate/src/average.rs - Foreign interface
datafusion/ffi/src/query_planner.rs - Substrait exchange
datafusion/substrait/src/logical_plan/consumer/expr/subquery.rs
Other components
- Query user
Connections 23
- Query user → SQL API: submits SQL
README.md - Query user → DataFrame API: builds queries
README.md - Query user → Command-line interface: runs queries
README.md - SQL API → SQL planner: plans SQL
datafusion/sql/src/query.rs - DataFrame API → Logical plans: builds plans
- SQL planner → Logical plans: produces plans
README.md - Logical plans → Logical optimizer: optimizes plans
README.md - Logical optimizer → Subquery rewrites: dispatches rules
datafusion/optimizer/src/optimizer.rs - Logical optimizer → Scalar rewrites: dispatches rules
datafusion/optimizer/src/optimizer.rs - Logical optimizer → Physical planner: plans execution
README.md - Physical planner → Query execution: creates plans
README.md - Query execution → Data sources: reads data
README.md - Data sources → File formats: reads formats
README.md - Query execution → Scalar subqueries: executes subqueries
datafusion/physical-plan/src/scalar_subquery.rs - Spill management → Spill files: creates spill files
datafusion/physical-plan/src/spill/spill_manager.rs - Query execution → Spill management: spills batches
datafusion/physical-plan/src/spill/spill_manager.rs - Metadata caches → Query execution: caches metadata
- Logical plans → Catalogs: resolves tables
- Logical plans → Built-in functions: uses expressions
- Foreign interface → Physical planner: delegates planning
datafusion/ffi/src/query_planner.rs - Substrait exchange → Logical plans: imports plans
- Command-line interface → CLI context: uses context
- CLI context → CLI object storage: configures storage
datafusion-cli/src/cli_context.rs