# apache/arrow-datafusion architecture

> Architecture diagram of the GitHub repository apache/arrow-datafusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/arrow-datafusion
- Repository: https://github.com/apache/arrow-datafusion

## Overview

##### Main Components
The system architecture should be divided into these key components:

a) Core Engine Components:
- Query Planning Layer
- SQL Parser
- Logical Planner
- Query Optimizer
- Physical Planner
- Execution Layer
- Runtime Environment
- Physical Operators
- Memory Management
- Task Scheduling

b) Data Source Layer:
- File Formats (Parquet, CSV, JSON, Avro)
- Custom Data Sources
- Catalog Management

c) API Layer:
- SQL API
- DataFrame API
- Extension Points

##### Component Relationships

**[Client APIs] → [Query Planning Layer] → [Execution Layer] → [Data Sources]**
- ↑                        ↑

- [Optimizer Layer]         [Runtime Environment]

##### Detailed Component Breakdown

Subcomponents:

a) Query Planning Layer:
- SQL Parser (datafusion/sql)
- Logical Plan Builder (datafusion/expr)
- Optimizer Rules (datafusion/optimizer)
- Physical Plan Generator (datafusion/physical-plan)

b) Execution Layer:
- Memory Pool Management (datafusion/execution)
- Task Scheduling (datafusion/execution)
- Physical Operators (datafusion/physical-expr)
- Runtime Environment (datafusion/common-runtime)

c) Data Sources:
- File Format Readers/Writers
- Table Providers
- Catalog Management

##### Extension Points

Extension capabilities:
- Custom Functions (UDF, UDAF, UDTF)
- Custom Data Sources
- Custom Optimizations
- Custom Physical Operators

##### Cross-Cutting Concerns

Aspects:
- Memory Management
- Error Handling
- Metrics Collection
- Configuration Management

##### Integration Points

External integration capabilities:
- Python Bindings
- Custom Extensions
- External Catalogs

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Client["Client Layer"]
        SQL["SQL API"]
        DF["DataFrame API"]:::api
        click DF "https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/src/dataframe/"
    end

    subgraph Planning["Query Planning Layer"]
        Parser["SQL Parser"]:::core
        click Parser "https://github.com/apache/arrow-datafusion/tree/main/datafusion/sql/"
        LogicalPlanner["Logical Planner"]:::core
        click LogicalPlanner "https://github.com/apache/arrow-datafusion/tree/main/datafusion/expr/"
        Optimizer["Query Optimizer"]:::optimizer
        click Optimizer "https://github.com/apache/arrow-datafusion/tree/main/datafusion/optimizer/"
        PhysicalPlanner["Physical Planner"]:::core
        click PhysicalPlanner "https://github.com/apache/arrow-datafusion/tree/main/datafusion/physical-plan/"
    end

    subgraph Execution["Execution Layer"]
        Runtime["Runtime Environment"]:::runtime
        click Runtime "https://github.com/apache/arrow-datafusion/tree/main/datafusion/execution/"
        PhysOps["Physical Operators"]:::core
        click PhysOps "https://github.com/apache/arrow-datafusion/tree/main/datafusion/physical-expr/"
        MemMgmt["Memory Management"]:::runtime
        click MemMgmt "https://github.com/apache/arrow-datafusion/tree/main/datafusion/execution/src/memory_pool/"
        TaskSched["Task Scheduling"]:::runtime
        click TaskSched "https://github.com/apache/arrow-datafusion/blob/main/datafusion/execution/src/task.rs"
    end

    subgraph DataSources["Data Source Layer"]
        FileFormats["File Format Handlers"]:::datasource
        click FileFormats "https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/src/datasource/file_format/"
        DataSrc["Data Sources"]:::datasource
        click DataSrc "https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/src/datasource/"
        Catalog["Catalog Management"]:::datasource
        click Catalog "https://github.com/apache/arrow-datafusion/tree/main/datafusion/catalog/"
    end

    subgraph Extensions["Extension Points"]
        CustomFuncs["Custom Functions"]
        subgraph Functions["Built-in Functions"]
            Window["Window Functions"]:::extension
            click Window "https://github.com/apache/arrow-datafusion/tree/main/datafusion/functions-window/"
            Agg["Aggregate Functions"]:::extension
            click Agg "https://github.com/apache/arrow-datafusion/tree/main/datafusion/functions-aggregate/"
            Table["Table Functions"]:::extension
            click Table "https://github.com/apache/arrow-datafusion/tree/main/datafusion/functions-table/"
        end
        ExtPoint["Extension Points"]:::extension
        click ExtPoint "https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/physical_plan/extension.rs"
    end

    subgraph Common["Common Components"]
        ExprEngine["Expression Engine"]:::core
        click ExprEngine "https://github.com/apache/arrow-datafusion/blob/main/datafusion/expr/src/expr.rs"
        CommonRuntime["Common Runtime"]:::runtime
        click CommonRuntime "https://github.com/apache/arrow-datafusion/tree/main/datafusion/common-runtime/"
    end

    SQL & DF --> Parser
    Parser --> LogicalPlanner
    LogicalPlanner --> Optimizer
    Optimizer --> PhysicalPlanner
    PhysicalPlanner --> Runtime
    Runtime --> PhysOps
    PhysOps --> DataSrc
    Runtime --> MemMgmt
    Runtime --> TaskSched
    DataSrc --> FileFormats
    DataSrc --> Catalog
    ExtPoint --> PhysicalPlanner
    CustomFuncs --> LogicalPlanner
    ExprEngine --> LogicalPlanner
    CommonRuntime --> Runtime

    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef datasource fill:#4CAF50,stroke:#2E7D32,color:white
    classDef api fill:#FFC107,stroke:#FFA000,color:black
    classDef extension fill:#FF9800,stroke:#F57C00,color:white
    classDef optimizer fill:#9C27B0,stroke:#7B1FA2,color:white
    classDef runtime fill:#757575,stroke:#424242,color:white

    subgraph Legend
        CoreComp["Core Component"]:::core
        DataComp["Data Source Component"]:::datasource
        APIComp["API Component"]:::api
        ExtComp["Extension Component"]:::extension
        OptComp["Optimizer Component"]:::optimizer
        RunComp["Runtime Component"]:::runtime
    end
```
