# satwikmishra11/datafusion architecture

> Architecture diagram of the GitHub repository satwikmishra11/datafusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/satwikmishra11/datafusion
- Repository: https://github.com/satwikmishra11/datafusion

## Overview

##### Main components
- Client/API Layer: This includes frontends such as the DataFusion CLI (datafusion-cli), the SQL and DataFrame APIs (exposed by modules in datafusion/core, datafusion/expr, datafusion/sql) and even external bindings like the Python DataFusion and FFI components.
- Query Processing & Planning Layer: Break this layer into components that parse SQL (and other query languages), build a Logical Plan (datafusion/expr, datafusion/sql), and then optimize it via transformation rules (optimizer components under datafusion/optimizer).
- Data Sources and Connectors: Highlight support for various formats (CSV, Parquet, Avro, JSON) as indicated by modules such as datasource-csv, datasource-parquet, datasource-avro, datasource-json, and the generic datasource interfaces.
- Support and Utility Components: This includes common runtime libraries (datafusion/common-runtime), in-memory types and utilities (datafusion/common), catalog functionality (datafusion/catalog, catalog-listing), and function modules for scalar, aggregate, window, nested, and table functions (functions-aggregate, functions-nested, functions-window, functions-table, functions overall).
- Extension/Integration Points: Represent the FFI module, substrait serialization (datafusion/substrait), and the “glue” components that allow integration with other systems (for example, the CLI and examples that illustrate how to extend or integrate the engine).

##### Relationships and data flows
- The Execution Engine then takes the physical plan and runs it across available compute resources, interfacing with various data source connectors to read/write data.

##### Architectural patterns and design principles
- Layered architecture that separates the query interface, planning (logical and physical), and execution.
- Extensibility points—the modular design allows plug‑in of new data sources or custom expressions/function implementations.

##### Using relevant technologies
- Presence of supporting tools for benchmarking, testing, and continuous integration (benchmarks, ci, etc.), though these might be peripheral for the core system design.
- Optional integrations like FFI, substrait, and CLI are important to show how the core engine is accessible through different interfaces.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client/API Layer"
        DFCLI("DataFusion CLI"):::frontend
        DFSQL("DataFusion SQL"):::frontend
        DFExpr("DataFusion Expr"):::frontend
    end

    subgraph "Query Processing & Logical Planning"
        SQLPars("SQL Parsing (datafusion/sql/src/parser.rs)"):::query
        LogPlan("Logical Plan Building (datafusion/expr/src/logical_plan)"):::query
    end

    subgraph "Optimizer"
        Optim("Optimizer (datafusion/optimizer)"):::optimizer
    end

    subgraph "Physical Planning & Execution Engine"
        PhysPlan("Physical Planner (datafusion/physical-plan)"):::execution
        ExecEngine("Execution Engine (datafusion/execution)"):::execution
    end

    subgraph "Data Sources & Connectors"
        CSV("CSV Connector"):::datasource
        Parquet("Parquet Connector"):::datasource
        Avro("Avro Connector"):::datasource
        JSON("JSON Connector"):::datasource
        GenSrc("Generic Data Source Interface"):::datasource
    end

    subgraph "Support & Utility Components"
        Common("Common Utilities (datafusion/common)"):::support
        Runtime("Common Runtime (datafusion/common-runtime)"):::support
        Catalog("Catalog (datafusion/catalog)"):::support
    end

    subgraph "Extension & Integration Points"
        FFI("FFI Module (datafusion/ffi)"):::extension
        Substrait("Substrait (datafusion/substrait)"):::extension
        Integration("Integration Examples (datafusion-examples)"):::extension
    end

    %% Connections between layers
    DFCLI --> DFSQL
    DFCLI --> DFExpr
    DFSQL --> SQLPars
    DFExpr --> LogPlan
    SQLPars --> Optim
    LogPlan --> Optim
    Optim --> PhysPlan
    PhysPlan --> ExecEngine
    ExecEngine --> CSV
    ExecEngine --> Parquet
    ExecEngine --> Avro
    ExecEngine --> JSON
    ExecEngine --> GenSrc
    ExecEngine -->|"results"| DFCLI

    %% Support & Utility integration (dashed lines)
    Common -.-> Optim
    Runtime -.-> Optim
    Catalog -.-> Optim
    Common -.-> ExecEngine
    Runtime -.-> ExecEngine
    Catalog -.-> ExecEngine

    %% Extension & Integration connections (dashed lines)
    DFSQL -.-> Integration
    DFCLI -.-> Integration
    FFI -.-> ExecEngine
    Substrait -.-> PhysPlan

    %% Click Events
    click DFCLI "https://github.com/satwikmishra11/datafusion/tree/main/datafusion-cli/src"
    click DFSQL "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/sql/src"
    click DFExpr "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/expr/src"
    click SQLPars "https://github.com/satwikmishra11/datafusion/blob/main/datafusion/sql/src/parser.rs"
    click LogPlan "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/expr/src/logical_plan"
    click Optim "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/optimizer/src"
    click PhysPlan "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/physical-plan/src"
    click ExecEngine "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/execution/src"
    click CSV "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/datasource-csv"
    click Parquet "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/datasource-parquet"
    click Avro "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/datasource-avro"
    click JSON "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/datasource-json"
    click GenSrc "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/datasource/src"
    click Common "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/common/src"
    click Runtime "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/common-runtime/src"
    click Catalog "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/catalog/src"
    click FFI "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/ffi/src"
    click Substrait "https://github.com/satwikmishra11/datafusion/tree/main/datafusion/substrait/src"
    click Integration "https://github.com/satwikmishra11/datafusion/tree/main/datafusion-examples/examples"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef query fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef optimizer fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef execution fill:#FFFACD,stroke:#000,stroke-width:2px;
    classDef datasource fill:#FFDAB9,stroke:#000,stroke-width:2px;
    classDef support fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef extension fill:#F08080,stroke:#000,stroke-width:2px;
```
