How vortex-data/vortex works
Project Type and Purpose
• This is an extensible, high-performance columnar file format toolkit (called “Vortex”) implemented as a Rust monorepo.
• It provides a core library (Rust crates) for logical schema, physical layout, encoding/decoding, compute kernels, I/O, query operators, plus language bindings (Python, Java, C++, Spark) and tooling (benchmarks, fuzzing, CLI TUI).
Top-Level Architecture from File Tree and README
• Core Rust crates under root:
• vortex (session, high-level API)
• vortex-dtype (logical schema, logical types)
• vortex-layout (physical storage layouts)
• vortex-file (file I/O, footer, segments)
• encodings/* (pluggable compression/encodings: ALP, RLE, dictionary, etc.)
• vortex-scan (vectorized scan operators)
• vortex-expr (expression trees, analysis)
• vortex-io (async I/O drivers, object store)
• vortex-ipc (IPC messages)
• Extensions and integrations:
• vortex-ffi, vortex-cxx, vortex-jni (C, C++, Java bindings)
• vortex-python, benchmarks-website (Python package and web UI)
• java/vortex-spark (Spark DataSource V2)
• vortex-duckdb (DuckDB table function integration)
• Auxiliary: docs/, fuzz/, bench-vortex/, xtask/, vortex-tui (CLI UI), scripts/, workflows.
Key Architectural Layers
a. Logical Layer
• Crates: vortex-dtype (logical schema), vortex-expr (expressions), vortex-scalar
b. Physical Layer
• Layout & Encoding: vortex-layout + encodings/*
• Storage I/O: vortex-file (footer, segments), vortex-io (object store drivers), vortex-ipc
c. Compute Layer
• Scan & operators: vortex-scan
d. Integration/Bindings Layer
• C/C++: vortex-ffi → C API, vortex-cxx → C++ API
• Java: vortex-jni → JNI wrappers, vortex-spark → Spark DataSource
• Python: vortex-python → PyO3 bindings, clickbench.py for benchmarks
e. Tooling & Infrastructure
• CLI: vortex-tui (“vx” browse)
• Benchmarks: bench-vortex + benchmarks-website
• Fuzz: fuzz/ crates
• CI/CD: GitHub workflows, scripts, xtask for codegen (FBS, Proto).
Relationships and Data Flows
• Data ingestion: object store (S3, GCS, local FS) ← vortex-io → vortex-file → vortex-array
• Logical planning: vortex-expr generates expression trees → vortex-scan executes scan over arrays
• Encoding/decoding: vortex-array invokes encodings/* via physical layer → layout writes via vortex-file
• Bindings: language clients (Python, Java, C++) call into core Rust via FFI/JNI/CXX → return arrays/records
• Query engines: DataFusion, DuckDB, Spark plug in via respective connectors → call into vortex-file + compute layers.
Architectural Patterns and Principles
• Layered modular design separating logical, physical, compute, and integration concerns.
• Pluggable encodings—each encoding lives in its own crate, selectable at runtime/compile time.
• FBS/Proto codegen for serialization—xtask generates definitions.
• Language-neutral API achieved via C-ABI, JNI, PyO3, CXX crates.
Technologies, Frameworks, and Libraries
• Rust ecosystem: Cargo workspaces, Tokio/async, Flatbuffers (.fbs), Protobuf (.proto), PyO3
• Build/tools: GitHub Actions, cargo, gradle (Java), uv (Rust/Python workspace), mimalloc optional.