# wagjamin/inkfuse architecture

> Architecture diagram of the GitHub repository wagjamin/inkfuse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wagjamin/inkfuse
- Repository: https://github.com/wagjamin/inkfuse

## Overview

##### Project type and purpose
- InkFuse is a research‐prototype, in‐memory, vectorized database engine that fuses code generation and interpretation.
- It does *not* include a SQL parser or optimizer; instead users supply hand‐crafted physical operator trees.
- Key directories (`algebra`, `codegen`, `exec`, `interpreter`, `runtime`, `storage`) confirm a layered engine architecture.

##### File structure
- Top‐level `src/` contains the core modules:
- `algebra`: transforms physical operator trees into a suboperator IR.
- `codegen`: builds C code artifacts (via JIT Clang) from the IR.
- `exec`: runs pipelines either by compiling or interpreting the suboperators.
- `interpreter`: generates a vectorized interpreter from the suboperators (fallback to non‐JIT).
- `runtime`: contains data‐structure implementations accessible from JIT code (hash tables, memory allocators).
- `storage`: simple columnar table storage (vectors of raw data).

##### README for additional insights
- Confirms the six core modules.
- Describes the “Incremental Fusion” concept: operators are fused at pipeline granularity for codegen or interpreted.
- Mentions CMake flags for enabling coverage or selecting custom JIT compilers.

##### Main components and their relationships
a. Main components:
1. Frontend Interface (user constructs physical operator trees)
2. Algebra Layer (`src/algebra` → suboperator IR)
3. Code Generator (`src/codegen` → C code + JIT compile)
4. Interpreter Generator (`src/interpreter` → vectorized interpreter)
5. Execution Engine (`src/exec` with CompiledRunner, InterpretedRunner, PipelineRunner)
6. Runtime Library (`src/runtime`: hash tables, memory)
7. Storage Engine (`src/storage`: columnar relations)
b. Data flow / interactions:
- User → Physical Operators → Algebra → Suboperator DAG
- Suboperator DAG → either → Codegen → Native Module → CompiledRunner
→ Interpreter → Vectorized Interpreter → InterpretedRunner
- Both runners use the Runtime Library APIs to build hash tables, manage memory.
- Both runners read/write data in Storage (columnar relations)
c. Architectural patterns & design principles:
- Pipeline‐based execution (DAG of suboperators)
- Fusion of operators at compile‐time (via JIT) or run‐time (via interpreter)
- Separation of concerns: algebraic rewriting, code generation, execution, storage
d. Key technologies/frameworks:
- Language: C++17 (CMake build system)
- JIT: clang-14 (configurable via CMake or CUSTOM_JIT env)
- Testing: GoogleTest, custom benchmarks

##### Elements
- Clear block labels for each module (e.g., “Algebra Layer”, “Code Generator”, “Runtime Library”).
- From User → Algebra
- Algebra → {Codegen, Interpreter}
- Codegen → CompiledRunner → Runtime Library & Storage
- Interpreter → InterpretedRunner → Runtime Library & Storage
- Blue rectangles for core engine modules (`algebra`, `exec`, `storage`, `runtime`)
- Red for codegen paths (`codegen`, JIT compiler)
- Green for interpretation path (`interpreter`, interpreter runner)
- Optionally annotate:
- CMake flags (`WITH_COVERAGE`, `JIT_CLANG_14`) near the build layer
- Bench/tools/test modules off to the side or bottom as non‐functional overlays.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: User Interface
    subgraph "1. User Interface"
        direction TB
        User["Frontend Interface\n(Physical Operator Trees)"]:::core
    end

    %% Layer 2: Algebra & IR
    subgraph "2. Algebra & IR"
        direction TB
        Algebra["Algebra Layer\n(Suboperator IR)"]:::core
    end

    %% Layer 3: Codegen & Interpreter
    subgraph "3. Codegen & Interpreter"
        direction TB
        Codegen["Code Generator\n(C IR → C Code + JIT)"]:::codegen
        JIT["Clang JIT (clang-14)"]:::external
        InterpreterGen["Interpreter Generator\n(Vectorized Interpreter)"]:::interpreter
    end

    %% Layer 4: Execution Engine
    subgraph "4. Execution Engine"
        direction TB
        CompiledRunner["CompiledRunner"]:::core
        InterpretedRunner["InterpretedRunner"]:::interpreter
        PipelineRunner["PipelineRunner"]:::core
    end

    %% Layer 5: Runtime Library
    subgraph "5. Runtime Library"
        direction TB
        Runtime["Runtime Library\n(Hash tables, memory)"]:::runtime
    end

    %% Layer 6: Storage
    subgraph "6. Storage"
        direction TB
        Storage["Storage Engine\n(Columnar Relations)"]:::storage
    end

    %% Peripheral Modules
    subgraph "Peripheral Modules"
        direction TB
        BenchTest["bench/ test/ tools/ reproduce/"]:::bench
    end

    %% Connections
    User -->|supplies operators| Algebra
    Algebra -->|generates IR| Codegen
    Algebra -->|generates IR| InterpreterGen
    Codegen -->|builds artifacts| CompiledRunner
    InterpreterGen -->|builds interpreter| InterpretedRunner
    CompiledRunner -->|executes pipelines| PipelineRunner
    InterpretedRunner -->|executes pipelines| PipelineRunner
    PipelineRunner -->|uses APIs| Runtime
    PipelineRunner -->|reads/writes| Storage
    Codegen -.->|optional/custom JIT| JIT

    %% Click Events
    click User "https://github.com/wagjamin/inkfuse/blob/main/src/algebra/RelAlgOp.h"
    click Algebra "https://github.com/wagjamin/inkfuse/tree/main/src/algebra/"
    click Codegen "https://github.com/wagjamin/inkfuse/tree/main/src/codegen/"
    click InterpreterGen "https://github.com/wagjamin/inkfuse/tree/main/src/interpreter/"
    click CompiledRunner "https://github.com/wagjamin/inkfuse/blob/main/src/exec/runners/CompiledRunner.*"
    click InterpretedRunner "https://github.com/wagjamin/inkfuse/blob/main/src/exec/runners/InterpretedRunner.*"
    click PipelineRunner "https://github.com/wagjamin/inkfuse/blob/main/src/exec/runners/PipelineRunner.*"
    click Runtime "https://github.com/wagjamin/inkfuse/tree/main/src/runtime/"
    click Storage "https://github.com/wagjamin/inkfuse/blob/main/src/storage/Relation.h"

    %% Styles
    classDef core fill:#AED6F1,stroke:#21618C,color:#1B2631
    classDef codegen fill:#F5B7B1,stroke:#B03A2E,color:#641E16
    classDef interpreter fill:#ABEBC6,stroke:#1D8348,color:#145A32
    classDef runtime fill:#F9E79F,stroke:#B9770E,color:#6E2C00
    classDef storage fill:#D7BDE2,stroke:#6C3483,color:#1B2631
    classDef external fill:#E5E7E9,stroke:#909497,color:#1B2631,stroke-dasharray: 5 5
    classDef bench fill:#EAECEE,stroke:#566573,color:#1B2631,stroke-dasharray: 2 2
```
