How wagjamin/inkfuse works
Project type and purpose
• InkFuse is a research‐prototype, in‐memory, vectorized database engine that fuses code generation and interpretation.
• It does *not* include a SQL parser or optimizer; instead users supply hand‐crafted physical operator trees.
• Key directories (algebra, codegen, exec, interpreter, runtime, storage) confirm a layered engine architecture.
File structure
• Top‐level src/ contains the core modules:
• algebra: transforms physical operator trees into a suboperator IR.
• codegen: builds C code artifacts (via JIT Clang) from the IR.
• exec: runs pipelines either by compiling or interpreting the suboperators.
• interpreter: generates a vectorized interpreter from the suboperators (fallback to non‐JIT).
• runtime: contains data‐structure implementations accessible from JIT code (hash tables, memory allocators).
• storage: simple columnar table storage (vectors of raw data).
README for additional insights
• Confirms the six core modules.
• Describes the “Incremental Fusion” concept: operators are fused at pipeline granularity for codegen or interpreted.
• Mentions CMake flags for enabling coverage or selecting custom JIT compilers.
Main components and their relationships
a. Main components:
1. Frontend Interface (user constructs physical operator trees)
2. Algebra Layer (src/algebra → suboperator IR)
3. Code Generator (src/codegen → C code + JIT compile)
4. Interpreter Generator (src/interpreter → vectorized interpreter)
5. Execution Engine (src/exec with CompiledRunner, InterpretedRunner, PipelineRunner)
6. Runtime Library (src/runtime: hash tables, memory)
7. Storage Engine (src/storage: columnar relations)
b. Data flow / interactions:
• User → Physical Operators → Algebra → Suboperator DAG
• Suboperator DAG → either → Codegen → Native Module → CompiledRunner
→ Interpreter → Vectorized Interpreter → InterpretedRunner
• Both runners use the Runtime Library APIs to build hash tables, manage memory.
• Both runners read/write data in Storage (columnar relations)
c. Architectural patterns & design principles:
• Pipeline‐based execution (DAG of suboperators)
• Fusion of operators at compile‐time (via JIT) or run‐time (via interpreter)
• Separation of concerns: algebraic rewriting, code generation, execution, storage
d. Key technologies/frameworks:
• Language: C++17 (CMake build system)
• JIT: clang-14 (configurable via CMake or CUSTOM_JIT env)
• Testing: GoogleTest, custom benchmarks
Elements
• Clear block labels for each module (e.g., “Algebra Layer”, “Code Generator”, “Runtime Library”).
• From User → Algebra
• Algebra → {Codegen, Interpreter}
• Codegen → CompiledRunner → Runtime Library & Storage
• Interpreter → InterpretedRunner → Runtime Library & Storage
• Blue rectangles for core engine modules (algebra, exec, storage, runtime)
• Red for codegen paths (codegen, JIT compiler)
• Green for interpretation path (interpreter, interpreter runner)
• Optionally annotate:
• CMake flags (WITH_COVERAGE, JIT_CLANG_14) near the build layer
• Bench/tools/test modules off to the side or bottom as non‐functional overlays.