# facebookincubator/velox architecture

> Architecture diagram of the GitHub repository facebookincubator/velox, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookincubator/velox
- Repository: https://github.com/facebookincubator/velox

## Overview

##### Main Components
- Input/Planning: Velox expects a fully optimized query plan as input. Even though it does not provide a SQL parser or dataframe layer, there is a planning stage that integrates with external systems.
- Functions Engine: Includes scalar functions, aggregates, and window functions (supporting Presto and Spark semantics).
- Operator Layer: Implements relational operators such as scan, join (hash, merge, nested-loop), filter, projection, aggregation, and even unnest and exchange operators.
- I/O and Connectors: A set of interfaces for data input/output – supporting various file formats (ORC/DWRF, Parquet, Nimble) as well as storage adapters for systems like AWS S3, HDFS, GCS, etc.
- Resource Management: Handles computational resources through memory arenas, thread pools, spill management, and caching.
- Extensibility: Velox enables specialization and extension via custom types, special functions, operators, network serializers, and more.

##### Relationships
- Data flows from an external query plan (or orchestrator) into the planning module.
- The planner directs execution through an operator tree, where each operator interacts with the expression evaluation engine and invokes functionality from the functions engine.
- Below the operator level, the vector and type systems manage the in-memory columnar data representations.
- The resource management subsystem supports the execution of operators and expression evaluation by overseeing memory allocation, thread execution, and potential spill to disk.
- I/O components allow operators to interact with external file systems and data sources/sinks.

##### Architectural Patterns/Design Principles
- Modularity: Each component (expression evaluation, functions, operators, I/O, resource management) is separated into its own module, often in separate directory trees (e.g., “velox/core”, “velox/exec”, “velox/vector”).
- Extensibility: Velox’s architecture is built to be composable and easily integrated with external systems, allowing developers to add custom components at many extension points.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input and Planning"
        QPI["Query Plan Input"]:::input
        PPP["Plan Parser/Planner"]:::input
    end

    subgraph "Core Engine"
        EEE["Expression Evaluation Engine"]:::core
        FE["Functions Engine"]:::core
        OL["Operator Layer"]:::core
    end

    subgraph "Memory & Resource Management"
        MM["Resource Management"]:::resource
    end

    subgraph "I/O and External Connectors"
        DC["Data Connectors"]:::io
        SA["Storage Adapters"]:::io
    end

    subgraph "Type and Vector Systems"
        TD["Data Types & Converters"]:::type
        CR["Columnar Data Representations"]:::type
    end

    subgraph "Extensibility"
        EXT["Extensibility"]:::extensibility
    end

    %% Connections
    QPI --> PPP
    PPP --> OL
    OL --> EEE
    OL --> FE
    OL --> DC
    OL --> SA
    EEE --> MM
    FE --> MM
    OL --> MM
    EEE --> TD
    EEE --> CR
    OL --> TD
    OL --> CR
    EXT --> OL
    EXT --> EEE
    EXT --> FE
    EXT --> DC
    EXT --> SA

    %% Click Events
    click PPP "https://github.com/facebookincubator/velox/blob/main/velox/parse/QueryPlanner.cpp"
    click EEE "https://github.com/facebookincubator/velox/blob/main/velox/expression/Expr.cpp"
    click FE "https://github.com/facebookincubator/velox/tree/main/velox/functions/"
    click OL "https://github.com/facebookincubator/velox/blob/main/velox/exec/Operator.cpp"
    click MM "https://github.com/facebookincubator/velox/tree/main/velox/common/memory/"
    click DC "https://github.com/facebookincubator/velox/tree/main/velox/dwio/"
    click SA "https://github.com/facebookincubator/velox/tree/main/velox/connectors/"
    click TD "https://github.com/facebookincubator/velox/tree/main/velox/type/"
    click CR "https://github.com/facebookincubator/velox/tree/main/velox/vector/"
    click EXT "https://github.com/facebookincubator/velox/tree/main/velox/external/"

    %% Class Definitions with Colors
    classDef input fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef core fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef resource fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef io fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef type fill:#DDA0DD,stroke:#333,stroke-width:2px;
    classDef extensibility fill:#F0E68C,stroke:#333,stroke-width:2px;
```
