# project-tsurugi/jogasaki architecture

> Architecture diagram of the GitHub repository project-tsurugi/jogasaki, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/project-tsurugi/jogasaki
- Repository: https://github.com/project-tsurugi/jogasaki

## Overview

##### Main Components
- Client/API Interface – Represent the external API endpoints or command-line tools that clients use (the examples directory shows various CLI applications).
- Compiler/Plan Builder – A module that takes SQL queries and converts them into executable plans. This is evident from directories such as src/jogasaki/plan, src/jogasaki/plan/compiler.cpp, and prepared statement handling.
- Executor – The heart of the system that processes and executes job tasks. Notice the extensive executor-related directories (e.g., src/jogasaki/executor) divided into sub-modules such as batch, exchange, process, and IO.
- Scheduler – Responsible for task scheduling and managing execution resources. This is visible through the src/jogasaki/scheduler directory, with implementations for different scheduling models (serial, hybrid, stealing, etc.).
- Data Storage and KVS – Modules like src/jogasaki/kvs, src/jogasaki/datastore, and meta handling (src/jogasaki/meta) indicate the underlying storage and data management layer.
- Logging, Profiling, and Diagnostics – The project supports integrated instrumentation (for example, via Tracy, LIKWID, and gflags/glog). There are dedicated modules for logging (src/jogasaki/logging*) and the README specifically discusses custom logging settings and external log integration.

##### Relationships and Data Flow
- The client submits SQL queries or batch jobs which are processed first by the Compiler/Plan Builder.
- The resulting execution plan is then handed over to the Executor, which decomposes the execution into smaller work units.
- The Scheduler oversees dispatching these tasks, possibly across multiple threads or processes, to optimize resource usage.
- During execution, the Executor interacts with the storage layer (KVS, datastores, metadata repositories) and uses specialized modules for tasks such as record exchange or batch processing.
- Logging, error handling, and performance profiling are interwoven within all layers to monitor the operation and diagnose issues.

##### Important Architectural Patterns and Design Principles
- Layered Architecture: Separation between the API/plan compilation layer, execution engine, scheduler, and storage.
- Modular Design: Each major subsystem (e.g., executor modules, scheduling strategies, data access) has its own directory and encapsulated functionality.
- Plugin/Configurable Instrumentation: The build options (such as -DTRACY_ENABLE and -DLIKWID_ENABLE) show that optional instrumentation and performance options are injected without disturbing core business logic.
- Dependency Injection and Build-Time Configuration: The extensive use of CMake options and modular subdirectories (like cmake, third_party, tools) emphasize a flexible build system that can integrate external libraries and custom settings.

##### Elements
– Client/API Interface
– SQL Compiler/Plan Builder
– Executor (with subdivisions: batch, process, exchange, I/O, function modules)
– Scheduler (with different scheduling strategies)
– Data Storage and KVS module
– Logging/Diagnostics/Profiling components
– External Integration Modules
– From Client → Compiler/Plan Builder → Executor/Scheduler
– Between Executor and Data Storage
– Feedback loops for error handling and diagnostic logging
- Annotations that mention key technologies (C++17, CMake, third-party libraries, etc.) and build options (for performance profiling, logging customization).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Interaction Layer
    subgraph "Interaction Layer"
        A["Client/API Interface"]:::client
    end

    %% Core Processing Layer
    subgraph "Core Processing Layer"
        B["SQL Compiler/Plan Builder"]:::compiler
        C["Executor (batch,exchange,process,IO,function)"]:::executor
        D["Scheduler (serial,hybrid,stealing)"]:::scheduler
    end

    %% Data Management Layer
    subgraph "Data Management Layer"
        E["Data Storage & KVS"]:::storage
    end

    %% Instrumentation & Diagnostics
    subgraph "Instrumentation & Diagnostics"
        F["Logging/Profiling/Diagnostics"]:::diagnostics
    end

    %% External Integration Layer
    subgraph "External Integrations"
        G["Third Party Integration Modules"]:::external
    end

    %% Connections
    A -->|"submits_query"| B
    B -->|"generates_plan"| C
    C -->|"executes_tasks"| D
    C -->|"accesses_data"| E
    C -->|"reports_status"| F
    D -->|"dispatches_tasks"| C
    B -->|"leverages"| G
    C -->|"leverages"| G
    E -->|"file_IO_with"| G
    F -->|"monitors"| G

    %% Click Events
    click A "https://github.com/project-tsurugi/jogasaki/tree/master/examples/client_cli"
    click A "https://github.com/project-tsurugi/jogasaki/tree/master/examples/sql_cli"
    click A "https://github.com/project-tsurugi/jogasaki/tree/master/include/jogasaki/api"
    click B "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/plan"
    click B "https://github.com/project-tsurugi/jogasaki/tree/master/include/jogasaki/api"
    click C "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/executor"
    click D "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/scheduler"
    click E "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/kvs"
    click E "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/datastore"
    click E "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/meta"
    click F "https://github.com/project-tsurugi/jogasaki/tree/master/src/jogasaki/external_log"
    click G "https://github.com/project-tsurugi/jogasaki/tree/master/third_party"
    click G "https://github.com/project-tsurugi/jogasaki/tree/master/src/tateyama"
    click G "https://github.com/project-tsurugi/jogasaki/blob/master/docs/arrow-dump-file-format-ja.md"

    %% Styles
    classDef client fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef compiler fill:#C2EABD,stroke:#333,stroke-width:2px;
    classDef executor fill:#B0E0E6,stroke:#333,stroke-width:2px;
    classDef scheduler fill:#D8BFD8,stroke:#333,stroke-width:2px;
    classDef storage fill:#FFFACD,stroke:#333,stroke-width:2px;
    classDef diagnostics fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef external fill:#FFD700,stroke:#333,stroke-width:2px;
```
