How facebookincubator/velox works
Main Components
• Input/Planning: Velox expects a fully optimized query plan as input. Even though it does not provide a SQL parser or dataframe layer, there is a planning stage that integrates with external systems.
• Functions Engine: Includes scalar functions, aggregates, and window functions (supporting Presto and Spark semantics).
• Operator Layer: Implements relational operators such as scan, join (hash, merge, nested-loop), filter, projection, aggregation, and even unnest and exchange operators.
• I/O and Connectors: A set of interfaces for data input/output – supporting various file formats (ORC/DWRF, Parquet, Nimble) as well as storage adapters for systems like AWS S3, HDFS, GCS, etc.
• Resource Management: Handles computational resources through memory arenas, thread pools, spill management, and caching.
• Extensibility: Velox enables specialization and extension via custom types, special functions, operators, network serializers, and more.
Relationships
• Data flows from an external query plan (or orchestrator) into the planning module.
• The planner directs execution through an operator tree, where each operator interacts with the expression evaluation engine and invokes functionality from the functions engine.
• Below the operator level, the vector and type systems manage the in-memory columnar data representations.
• The resource management subsystem supports the execution of operators and expression evaluation by overseeing memory allocation, thread execution, and potential spill to disk.
• I/O components allow operators to interact with external file systems and data sources/sinks.
Architectural Patterns/Design Principles
• Modularity: Each component (expression evaluation, functions, operators, I/O, resource management) is separated into its own module, often in separate directory trees (e.g., “velox/core”, “velox/exec”, “velox/vector”).
• Extensibility: Velox’s architecture is built to be composable and easily integrated with external systems, allowing developers to add custom components at many extension points.