How anchi205/in-mem-store works
Main Components
• Backend Engine: This is the core storage engine implementation and is found inside the “backend/leanstore” directory. It includes:
• Core engine files (LeanStore.cpp/.hpp, Config files)
• Concurrency and recovery modules (inside the concurrency-recovery subdirectory) handling transaction processing, group commit mechanisms, logging, and WAL management.
• Profiling modules (found under profiling directories for counters and tables) that collect performance data.
• Storage layers, including specialized implementations like B-tree storage structures and buffer management (e.g., in the “storage/btree” and “storage/buffer-manager” directories).
• Synchronization primitives and threading support.
• Utility routines to support hash, file handling, parallelization, etc.
• Frontend/Examples: These modules (in the “frontend” directory) provide the clients or benchmarking interfaces to:
• Run micro-benchmarks, minimal examples, scaled experiments, and standard benchmarks (TPC-C, YCSB).
• Provide adapters to interface LeanStore with other storage systems (e.g., LMDB, RocksDB, WiredTiger).
• Libraries and Shared-headers: These directories house CMake modules, third-party integration scripts, and shared header definitions that are used across both backend and frontend parts.
Relationships and Interactions
• The Backend Engine operates as the core service that manages data storage, retrieval, concurrency, and recovery. Its different modules (concurrency, buffer management, storage, profiling) interact internally:
• The concurrency-recovery logic ensures transactions are processed and logged correctly.
• Buffer management and storage systems provide data persistence and optimization for in-memory operations.
• Profiling collects runtime metrics, affecting tuning and debugging.
• The Frontend acts as a series of clients or test harnesses that interface with the backend. They typically demonstrate:
• How to load data (bulk_loading in “mix”)
• How to perform transactions or run TPC-C/YCSB workloads against the engine.
• Adapters in the “shared” folder abstract different external storage engines, showing the modular and extensible nature of the engine.
• The “libs” and “shared-headers” provide supporting frameworks (e.g., Boost, RocksDB, YAML, spdlog) that the build system uses to link dependencies and standardize interfaces.
Architectural Patterns and Design Principles
• Modularity: The project is clearly modular, separating core storage and recovery logic from benchmark/testing and external adapter components.
• Layered Architecture: There is a clear separation between low-level storage management (buffer and storage managers) and higher-level transaction recovery and profiling.
• Extensibility: The frontend adapts the core engine through various adapters. This indicates an effort to allow LeanStore to integrate with multiple external data systems.
• Performance Orientation: The emphasis on a Virtual-Memory Assisted Buffer Pool and the usage of many-core and NVMe optimizations suggest that performance-critical design principles are employed.