# project-tsurugi/shirakami architecture

> Architecture diagram of the GitHub repository project-tsurugi/shirakami, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/project-tsurugi/shirakami
- Repository: https://github.com/project-tsurugi/shirakami

## Overview

- API Layer / Public Interface:
– Represented by files in include/shirakami (e.g., api_diagnostic.h, api_result.h, etc.).
– This layer is what external users will interact with (transaction and diagnostic APIs).
- Concurrency Control Engine:
– Located in src/concurrency_control and its subdirectories (bg_work for background tasks, long_tx, short_tx, read_only_tx, and various helper modules).
– Include internal mechanisms such as epoch management, garbage collection, session management, and conflict handling.
- Database Subsystem:
– Shown by src/database and related thread pool management (thread_pool.cpp, tx_state_notification.cpp) that orchestrate the operations.
- Datastore / Storage Engine:
– Found in src/datastore/limestone which includes datastore.cpp, limestone_api_helper, and lpwal implementation.
– This component acts as the underlying persistent storage; note the integration of parallel write-ahead logging (PWAL) and the dependency on Yakushima.
- Indexing Component:
– Represented in src/index/yakushima – for the in-memory indexing engine, used by the transaction engine.
- Benchmark and Testing Modules:
– Directories such as bench and test provide performance benchmarks and extensive unit and integration tests.
– Though not part of the runtime architecture, they help validate and fine-tune the engine.

##### Relationships and Data Flow
- How external calls flow from the API layer into the transaction engine:
– Clients make calls that are forwarded into the API, which then interacts with the concurrency control engine.
- Within the concurrency control component, delineate submodules:
– Indicate how transaction requests are processed with components like “epoch management,” “local set,” “ongoing_tx,” “read/short/long transactions” (each having their own modules).
– Illustrate background work (bg_commit and garbage collection) as auxiliary processes that support transaction finalization.
- Interaction with the database layer:
– The concurrency control engine sends transaction commands to the database subsystem that handles thread pooling and coordination.
- Connection to the datastore/storage engine:
– The database system interacts with datastore (limestone) to perform logging, update operations, and persistent storage.
– Mark the use of parallel write-ahead logging (enabled by BUILD_PWAL) and integration with external in-memory index (Yakushima).

##### Architectural Patterns and Design Principles
- Modularization: Separate directories for concurrency control, database access, and the datastore enforce separation of concerns.
- Dependency Injection / Configurability: The use of CMake options (e.g., for enabling/disabling tests, benchmark builds, coverage, logging configuration) shows that the engine is designed to be highly configurable.
- Scalability / Performance: Parameters such as DKVS_MAX_PARALLEL_THREADS and PARAM_RETRY_READ indicate tuning points for high concurrency and performance, which should be highlighted.
- Layered Architecture: Identify a clear separation from user-facing API, transaction/logic handling, and storage/persistence layers.

##### Relevant Technologies, Frameworks, and Tools
- C++17 as the language standard.
- CMake (with Ninja generator support) for build configuration.
- Yakushima for in-memory indexing.
- Limestone for the storage layer and parallel write-ahead logging.
- Testing frameworks (googletest in third_party and numerous test cases under test/).
- Clang tools (clang-format, clang-tidy) for code formatting and static analysis.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Nodes
    Clients(["Clients"]):::client

    %% API Layer
    API["API Layer<br/>(Public Interface)"]:::api

    %% Concurrency Control Engine Subgraph
    subgraph "Concurrency Control Engine"
        CC_Core["Core Engine"]:::transaction
        CC_Long["Long Transaction"]:::transaction
        CC_Short["Short Transaction"]:::transaction
        CC_Read["Read Only Transaction"]:::transaction
        CC_BG["Background Tasks<br/>(bg_work)"]:::transaction
    end

    %% Database Subsystem Subgraph
    subgraph "Database Subsystem"
        DB["Database Manager"]:::storage
        TP["Thread Pool &<br/>Notification Handler"]:::storage
    end

    %% Datastore / Storage Engine Subgraph
    subgraph "Datastore / Storage Engine"
        DS["Limestone Storage Engine<br/>(PWAL enabled)"]:::storage
    end

    %% Indexing Component
    Index["In-Memory Index<br/>(Yakushima)"]:::storage

    %% Benchmark and Testing Modules
    BenchTest["Benchmark & Testing<br/>(bench & test)"]:::testing

    %% Data Flow Connections
    Clients -->|"send request"| API
    API -->|"calls"| CC_Core
    API -->|"calls"| CC_Long
    API -->|"calls"| CC_Short
    API -->|"calls"| CC_Read
    API -->|"calls"| CC_BG

    %% Internal Concurrency Control flows
    CC_Core -->|"orchestrates"| CC_Long
    CC_Core -->|"orchestrates"| CC_Short
    CC_Core -->|"orchestrates"| CC_Read
    CC_Core -->|"triggers"| CC_BG

    %% Flow into Database Subsystem
    CC_Core -->|"submits TX"| DB
    CC_Long -->|"submits TX"| DB
    CC_Short -->|"submits TX"| DB
    CC_Read -->|"submits TX"| DB
    CC_BG -->|"finalizes TX"| DB

    %% Database internal flow
    DB -->|"manages tasks"| TP
    TP -->|"executes tasks"| DS

    %% Interaction with Datastore directly
    DB -->|"calls storage"| DS

    %% Indexing interaction from Concurrency Engine
    CC_Core -->|"queries/upd"| Index
    CC_Long -->|"queries/upd"| Index
    CC_Short -->|"queries/upd"| Index
    CC_Read -->|"queries/upd"| Index

    %% Benchmark and Testing feedback loop
    BenchTest ---|"validates"| API
    BenchTest ---|"validates"| CC_Core
    BenchTest ---|"validates"| DB

    %% Click Events
    click API "https://github.com/project-tsurugi/shirakami/tree/master/include/shirakami"
    click CC_Core "https://github.com/project-tsurugi/shirakami/tree/master/src/concurrency_control"
    click CC_BG "https://github.com/project-tsurugi/shirakami/tree/master/src/concurrency_control/bg_work"
    click DB "https://github.com/project-tsurugi/shirakami/tree/master/src/database"
    click TP "https://github.com/project-tsurugi/shirakami/tree/master/src/database"
    click DS "https://github.com/project-tsurugi/shirakami/tree/master/src/datastore/limestone"
    click Index "https://github.com/project-tsurugi/shirakami/tree/master/src/index/yakushima"
    click BenchTest "https://github.com/project-tsurugi/shirakami/tree/master/bench"

    %% Styles
    classDef api fill:#b3d9ff,stroke:#333,stroke-width:2px;
    classDef transaction fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef storage fill:#ffffcc,stroke:#333,stroke-width:2px;
    classDef testing fill:#e0e0e0,stroke:#333,stroke-width:2px;
    classDef client fill:#ffcccc,stroke:#333,stroke-width:2px;
```
