How project-tsurugi/jogasaki works
Main Components
• Client/API Interface – Represent the external API endpoints or command-line tools that clients use (the examples directory shows various CLI applications).
• Compiler/Plan Builder – A module that takes SQL queries and converts them into executable plans. This is evident from directories such as src/jogasaki/plan, src/jogasaki/plan/compiler.cpp, and prepared statement handling.
• Executor – The heart of the system that processes and executes job tasks. Notice the extensive executor-related directories (e.g., src/jogasaki/executor) divided into sub-modules such as batch, exchange, process, and IO.
• Scheduler – Responsible for task scheduling and managing execution resources. This is visible through the src/jogasaki/scheduler directory, with implementations for different scheduling models (serial, hybrid, stealing, etc.).
• Data Storage and KVS – Modules like src/jogasaki/kvs, src/jogasaki/datastore, and meta handling (src/jogasaki/meta) indicate the underlying storage and data management layer.
• Logging, Profiling, and Diagnostics – The project supports integrated instrumentation (for example, via Tracy, LIKWID, and gflags/glog). There are dedicated modules for logging (src/jogasaki/logging*) and the README specifically discusses custom logging settings and external log integration.
Relationships and Data Flow
• The client submits SQL queries or batch jobs which are processed first by the Compiler/Plan Builder.
• The resulting execution plan is then handed over to the Executor, which decomposes the execution into smaller work units.
• The Scheduler oversees dispatching these tasks, possibly across multiple threads or processes, to optimize resource usage.
• During execution, the Executor interacts with the storage layer (KVS, datastores, metadata repositories) and uses specialized modules for tasks such as record exchange or batch processing.
• Logging, error handling, and performance profiling are interwoven within all layers to monitor the operation and diagnose issues.
Important Architectural Patterns and Design Principles
• Layered Architecture: Separation between the API/plan compilation layer, execution engine, scheduler, and storage.
• Modular Design: Each major subsystem (e.g., executor modules, scheduling strategies, data access) has its own directory and encapsulated functionality.
• Plugin/Configurable Instrumentation: The build options (such as -DTRACY_ENABLE and -DLIKWID_ENABLE) show that optional instrumentation and performance options are injected without disturbing core business logic.
• Dependency Injection and Build-Time Configuration: The extensive use of CMake options and modular subdirectories (like cmake, third_party, tools) emphasize a flexible build system that can integrate external libraries and custom settings.
Elements
– Client/API Interface
– SQL Compiler/Plan Builder
– Executor (with subdivisions: batch, process, exchange, I/O, function modules)
– Scheduler (with different scheduling strategies)
– Data Storage and KVS module
– Logging/Diagnostics/Profiling components
– External Integration Modules
– From Client → Compiler/Plan Builder → Executor/Scheduler
– Between Executor and Data Storage
– Feedback loops for error handling and diagnostic logging
• Annotations that mention key technologies (C++17, CMake, third-party libraries, etc.) and build options (for performance profiling, logging customization).