# vortex-data/vortex architecture

> Architecture diagram of the GitHub repository vortex-data/vortex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vortex-data/vortex
- Repository: https://github.com/vortex-data/vortex

## Overview

##### Project Type and Purpose
- This is an extensible, high-performance columnar file format toolkit (called “Vortex”) implemented as a Rust monorepo.
- It provides a core library (Rust crates) for logical schema, physical layout, encoding/decoding, compute kernels, I/O, query operators, plus language bindings (Python, Java, C++, Spark) and tooling (benchmarks, fuzzing, CLI TUI).

##### Top-Level Architecture from File Tree and README
- Core Rust crates under root:
- vortex (session, high-level API)
- vortex-dtype (logical schema, logical types)
- vortex-layout (physical storage layouts)
- vortex-file (file I/O, footer, segments)
- encodings/* (pluggable compression/encodings: ALP, RLE, dictionary, etc.)
- vortex-scan (vectorized scan operators)
- vortex-expr (expression trees, analysis)
- vortex-io (async I/O drivers, object store)
- vortex-ipc (IPC messages)
- Extensions and integrations:
- vortex-ffi, vortex-cxx, vortex-jni (C, C++, Java bindings)
- vortex-python, benchmarks-website (Python package and web UI)
- java/vortex-spark (Spark DataSource V2)
- vortex-duckdb (DuckDB table function integration)
- Auxiliary: docs/, fuzz/, bench-vortex/, xtask/, vortex-tui (CLI UI), scripts/, workflows.

##### Key Architectural Layers
a. Logical Layer
- Crates: vortex-dtype (logical schema), vortex-expr (expressions), vortex-scalar
b. Physical Layer
- Layout & Encoding: vortex-layout + encodings/*
- Storage I/O: vortex-file (footer, segments), vortex-io (object store drivers), vortex-ipc
c. Compute Layer
- Scan & operators: vortex-scan
d. Integration/Bindings Layer
- C/C++: vortex-ffi → C API, vortex-cxx → C++ API
- Java: vortex-jni → JNI wrappers, vortex-spark → Spark DataSource
- Python: vortex-python → PyO3 bindings, clickbench.py for benchmarks
e. Tooling & Infrastructure
- CLI: vortex-tui (“vx” browse)
- Benchmarks: bench-vortex + benchmarks-website
- Fuzz: fuzz/ crates
- CI/CD: GitHub workflows, scripts, xtask for codegen (FBS, Proto).

##### Relationships and Data Flows
- Data ingestion: object store (S3, GCS, local FS) ← vortex-io → vortex-file → vortex-array
- Logical planning: vortex-expr generates expression trees → vortex-scan executes scan over arrays
- Encoding/decoding: vortex-array invokes encodings/* via physical layer → layout writes via vortex-file
- Bindings: language clients (Python, Java, C++) call into core Rust via FFI/JNI/CXX → return arrays/records
- Query engines: DataFusion, DuckDB, Spark plug in via respective connectors → call into vortex-file + compute layers.

##### Architectural Patterns and Principles
- Layered modular design separating logical, physical, compute, and integration concerns.
- Pluggable encodings—each encoding lives in its own crate, selectable at runtime/compile time.
- FBS/Proto codegen for serialization—xtask generates definitions.
- Language-neutral API achieved via C-ABI, JNI, PyO3, CXX crates.

##### Technologies, Frameworks, and Libraries
- Rust ecosystem: Cargo workspaces, Tokio/async, Flatbuffers (.fbs), Protobuf (.proto), PyO3
- Build/tools: GitHub Actions, cargo, gradle (Java), uv (Rust/Python workspace), mimalloc optional.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core API Layer
    subgraph "Core API" 
        direction TB
        CoreAPI["vortex\n(session & API)"]:::core
    end

    %% Logical Layer
    subgraph "Logical Layer"
        direction TB
        DType["vortex-dtype\n(logical schema)"]:::core
        Expr["vortex-expr\n(expression trees)"]:::core
        Scalar["vortex-scalar\n(scalar types)"]:::core
    end

    %% Physical Layer
    subgraph "Physical Layer"
        direction TB
        Layout["vortex-layout\n(physical layout)"]:::core
        Encodings["Encodings Plugin\n(ALP, RLE, dict, etc.)"]:::plugin
        FileIO["vortex-file\n(file I/O, footer, segments)"]:::core
        AsyncIO["vortex-io\n(async I/O, object store drivers)"]:::core
        IPC["vortex-ipc\n(IPC messaging)"]:::core
    end

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        Array["vortex-array\n(array kernels & Arrow)"]:::compute
        Scan["vortex-scan\n(vectorized scan)"]:::compute
    end

    %% Integration/Bindings Layer
    subgraph "Integration & Bindings"
        direction TB
        FFI["vortex-ffi\n(C FFI)"]:::integration
        CXX["vortex-cxx\n(C++ API)"]:::integration
        JNI["vortex-jni\n(JNI wrappers)"]:::integration
        Spark["vortex-spark\n(Spark DataSource V2)"]:::integration
        Python["vortex-python\n(PyO3 bindings)"]:::integration
        DuckDB["vortex-duckdb\n(DuckDB table func)"]:::integration
        DataFusion["vortex-datafusion\n(DataFusion connector)"]:::integration
    end

    %% Tooling & Infrastructure
    subgraph "Tooling & Infrastructure"
        direction TB
        CLI["vortex-tui\n(CLI TUI)"]:::tooling
        BenchRust["bench-vortex\n(Rust benchmarks)"]:::tooling
        BenchWeb["benchmarks-website\n(Web UI)"]:::tooling
        Fuzz["fuzz\n(Fuzz testing)"]:::tooling
        Codegen["xtask\n(Flatbuffers/Proto codegen)"]:::tooling
        ScriptsCI["scripts & workflows\n(CI/CD)"]:::tooling
    end

    %% External Systems
    ObjectStore["Object Store\n(S3, GCS, FS)"]:::external
    ArrowExt["Apache Arrow"]:::external
    DFExt["Apache DataFusion"]:::external
    DuckExt["DuckDB"]:::external
    SparkExt["Spark"]:::external
    PythonExt["Python Ecosystem\n(Pandas, Polars)"]:::external

    %% Data Flows & Relationships
    ObjectStore -->|ingest data| AsyncIO
    AsyncIO --> FileIO
    FileIO --> Array
    Array --> Scan
    Scan -->|results| CoreAPI

    CoreAPI --> DType
    CoreAPI --> Expr
    CoreAPI --> Scalar

    DType --> Expr
    Expr --> Scan

    Layout --> FileIO
    Encodings --> Layout
    Encodings --> Array

    Array -->|zero-copy buffers| ArrowExt
    Scan --> DFExt
    FileIO --> DuckExt
    Spark --> SparkExt
    Python --> PythonExt

    FFI --> CoreAPI
    CXX --> CoreAPI
    JNI --> CoreAPI
    Python --> CoreAPI
    DuckDB --> CoreAPI
    DataFusion --> CoreAPI

    CLI --> CoreAPI
    BenchRust --> CoreAPI
    BenchWeb --> CoreAPI
    Fuzz --> CoreAPI
    Codegen --> CoreAPI
    ScriptsCI --> CoreAPI

    %% Click Events
    click CoreAPI "https://github.com/vortex-data/vortex/blob/develop/vortex/src/lib.rs"
    click CoreAPI "https://github.com/vortex-data/vortex/blob/develop/vortex/src/session.rs"
    click DType "https://github.com/vortex-data/vortex/blob/develop/vortex-dtype/src/dtype.rs"
    click Expr "https://github.com/vortex-data/vortex/blob/develop/vortex-expr/src/lib.rs"
    click Scalar "https://github.com/vortex-data/vortex/blob/develop/vortex-scalar/src/scalar.rs"
    click Layout "https://github.com/vortex-data/vortex/blob/develop/vortex-layout/src/layout.rs"
    click Encodings "https://github.com/vortex-data/vortex/tree/develop/encodings/"
    click FileIO "https://github.com/vortex-data/vortex/blob/develop/vortex-file/src/file.rs"
    click FileIO "https://github.com/vortex-data/vortex/tree/develop/vortex-file/src/footer"
    click AsyncIO "https://github.com/vortex-data/vortex/blob/develop/vortex-io/src/file/driver.rs"
    click AsyncIO "https://github.com/vortex-data/vortex/blob/develop/vortex-io/src/file/object_store.rs"
    click IPC "https://github.com/vortex-data/vortex/tree/develop/vortex-ipc/src/messages"
    click Array "https://github.com/vortex-data/vortex/blob/develop/vortex-array/src/lib.rs"
    click Array "https://github.com/vortex-data/vortex/tree/develop/vortex-array/src/compute"
    click Scan "https://github.com/vortex-data/vortex/blob/develop/vortex-scan/src/lib.rs"
    click FFI "https://github.com/vortex-data/vortex/blob/develop/vortex-ffi/src/lib.rs"
    click CXX "https://github.com/vortex-data/vortex/blob/develop/vortex-cxx/cpp/include/vortex.hpp"
    click CXX "https://github.com/vortex-data/vortex/tree/develop/vortex-cxx/cpp/src/"
    click JNI "https://github.com/vortex-data/vortex/blob/develop/vortex-jni/src/lib.rs"
    click JNI "https://github.com/vortex-data/vortex/tree/develop/java/vortex-jni/src/main/java/dev/vortex/api"
    click Spark "https://github.com/vortex-data/vortex/blob/develop/java/vortex-spark/src/main/java/dev/vortex/spark/VortexDataSourceV2.java"
    click Python "https://github.com/vortex-data/vortex/blob/develop/vortex-python/src/lib.rs"
    click Python "https://github.com/vortex-data/vortex/blob/develop/vortex-python/python/vortex/__init__.py"
    click DuckDB "https://github.com/vortex-data/vortex/blob/develop/vortex-duckdb/src/lib.rs"
    click DuckDB "https://github.com/vortex-data/vortex/tree/develop/vortex-duckdb/cpp/include/duckdb_vx"
    click DataFusion "https://github.com/vortex-data/vortex/blob/develop/vortex-datafusion/src/lib.rs"
    click CLI "https://github.com/vortex-data/vortex/blob/develop/vortex-tui/src/main.rs"
    click BenchRust "https://github.com/vortex-data/vortex/blob/develop/bench-vortex/Cargo.toml"
    click BenchRust "https://github.com/vortex-data/vortex/blob/develop/bench-vortex/src/bench_run.rs"
    click BenchWeb "https://github.com/vortex-data/vortex/blob/develop/benchmarks-website/index.html"
    click BenchWeb "https://github.com/vortex-data/vortex/tree/develop/benchmarks-website/"
    click Fuzz "https://github.com/vortex-data/vortex/blob/develop/fuzz/Cargo.toml"
    click Fuzz "https://github.com/vortex-data/vortex/tree/develop/fuzz/fuzz_targets/"
    click Codegen "https://github.com/vortex-data/vortex/blob/develop/xtask/src/generate_fbs.rs"
    click Codegen "https://github.com/vortex-data/vortex/blob/develop/xtask/src/generate_proto.rs"
    click ScriptsCI "https://github.com/vortex-data/vortex/tree/develop/scripts/"
    click ScriptsCI "https://github.com/vortex-data/vortex/tree/develop/.github/workflows/"

    %% Styles
    classDef core fill:#cfe2f3,stroke:#2f75b5,color:#000;
    classDef compute fill:#d9ead3,stroke:#6aa84f,color:#000;
    classDef integration fill:#ffe599,stroke:#bf9000,color:#000;
    classDef tooling fill:#eeeeee,stroke:#888888,color:#000;
    classDef external fill:#f4cccc,stroke:#cc0000,color:#000;
    classDef plugin fill:#fff2cc,stroke:#b59f3b,color:#000;
```
