How project-tsurugi/shirakami works
• API Layer / Public Interface:
– Represented by files in include/shirakami (e.g., api_diagnostic.h, api_result.h, etc.).
– This layer is what external users will interact with (transaction and diagnostic APIs).
• Concurrency Control Engine:
– Located in src/concurrency_control and its subdirectories (bg_work for background tasks, long_tx, short_tx, read_only_tx, and various helper modules).
– Include internal mechanisms such as epoch management, garbage collection, session management, and conflict handling.
• Database Subsystem:
– Shown by src/database and related thread pool management (thread_pool.cpp, tx_state_notification.cpp) that orchestrate the operations.
• Datastore / Storage Engine:
– Found in src/datastore/limestone which includes datastore.cpp, limestone_api_helper, and lpwal implementation.
– This component acts as the underlying persistent storage; note the integration of parallel write-ahead logging (PWAL) and the dependency on Yakushima.
• Indexing Component:
– Represented in src/index/yakushima – for the in-memory indexing engine, used by the transaction engine.
• Benchmark and Testing Modules:
– Directories such as bench and test provide performance benchmarks and extensive unit and integration tests.
– Though not part of the runtime architecture, they help validate and fine-tune the engine.
Relationships and Data Flow
• How external calls flow from the API layer into the transaction engine:
– Clients make calls that are forwarded into the API, which then interacts with the concurrency control engine.
• Within the concurrency control component, delineate submodules:
– Indicate how transaction requests are processed with components like “epoch management,” “local set,” “ongoing_tx,” “read/short/long transactions” (each having their own modules).
– Illustrate background work (bg_commit and garbage collection) as auxiliary processes that support transaction finalization.
• Interaction with the database layer:
– The concurrency control engine sends transaction commands to the database subsystem that handles thread pooling and coordination.
• Connection to the datastore/storage engine:
– The database system interacts with datastore (limestone) to perform logging, update operations, and persistent storage.
– Mark the use of parallel write-ahead logging (enabled by BUILD_PWAL) and integration with external in-memory index (Yakushima).
Architectural Patterns and Design Principles
• Modularization: Separate directories for concurrency control, database access, and the datastore enforce separation of concerns.
• Dependency Injection / Configurability: The use of CMake options (e.g., for enabling/disabling tests, benchmark builds, coverage, logging configuration) shows that the engine is designed to be highly configurable.
• Scalability / Performance: Parameters such as DKVS_MAX_PARALLEL_THREADS and PARAM_RETRY_READ indicate tuning points for high concurrency and performance, which should be highlighted.
• Layered Architecture: Identify a clear separation from user-facing API, transaction/logic handling, and storage/persistence layers.
Relevant Technologies, Frameworks, and Tools
• C++17 as the language standard.
• CMake (with Ninja generator support) for build configuration.
• Yakushima for in-memory indexing.
• Limestone for the storage layer and parallel write-ahead logging.
• Testing frameworks (googletest in third_party and numerous test cases under test/).
• Clang tools (clang-format, clang-tidy) for code formatting and static analysis.