# anchi205/in-mem-store architecture

> Architecture diagram of the GitHub repository anchi205/in-mem-store, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anchi205/in-mem-store
- Repository: https://github.com/anchi205/in-mem-store

## Overview

##### Main Components
- Backend Engine: This is the core storage engine implementation and is found inside the “backend/leanstore” directory. It includes:
- Core engine files (LeanStore.cpp/.hpp, Config files)
- Concurrency and recovery modules (inside the concurrency-recovery subdirectory) handling transaction processing, group commit mechanisms, logging, and WAL management.
- Profiling modules (found under profiling directories for counters and tables) that collect performance data.
- Storage layers, including specialized implementations like B-tree storage structures and buffer management (e.g., in the “storage/btree” and “storage/buffer-manager” directories).
- Synchronization primitives and threading support.
- Utility routines to support hash, file handling, parallelization, etc.
- Frontend/Examples: These modules (in the “frontend” directory) provide the clients or benchmarking interfaces to:
- Run micro-benchmarks, minimal examples, scaled experiments, and standard benchmarks (TPC-C, YCSB).
- Provide adapters to interface LeanStore with other storage systems (e.g., LMDB, RocksDB, WiredTiger).
- Libraries and Shared-headers: These directories house CMake modules, third-party integration scripts, and shared header definitions that are used across both backend and frontend parts.

##### Relationships and Interactions
- The Backend Engine operates as the core service that manages data storage, retrieval, concurrency, and recovery. Its different modules (concurrency, buffer management, storage, profiling) interact internally:
- The concurrency-recovery logic ensures transactions are processed and logged correctly.
- Buffer management and storage systems provide data persistence and optimization for in-memory operations.
- Profiling collects runtime metrics, affecting tuning and debugging.
- The Frontend acts as a series of clients or test harnesses that interface with the backend. They typically demonstrate:
- How to load data (bulk_loading in “mix”)
- How to perform transactions or run TPC-C/YCSB workloads against the engine.
- Adapters in the “shared” folder abstract different external storage engines, showing the modular and extensible nature of the engine.
- The “libs” and “shared-headers” provide supporting frameworks (e.g., Boost, RocksDB, YAML, spdlog) that the build system uses to link dependencies and standardize interfaces.

##### Architectural Patterns and Design Principles
- Modularity: The project is clearly modular, separating core storage and recovery logic from benchmark/testing and external adapter components.
- Layered Architecture: There is a clear separation between low-level storage management (buffer and storage managers) and higher-level transaction recovery and profiling.
- Extensibility: The frontend adapts the core engine through various adapters. This indicates an effort to allow LeanStore to integrate with multiple external data systems.
- Performance Orientation: The emphasis on a Virtual-Memory Assisted Buffer Pool and the usage of many-core and NVMe optimizations suggest that performance-critical design principles are employed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Engine Subgraph
    subgraph "Core Engine"
        CS("Core Storage Engine"):::core
        CR("Concurrency & Recovery Module"):::core
        PR("Profiling Module"):::core
        subgraph "Storage Module"
            BT("B-Tree Structures"):::storage
            BM("Buffer Management"):::storage
        end
        subgraph "Support Utilities"
            SY("Synchronization"):::util
            TH("Threading"):::util
            UT("Utilities & Supporting Routines"):::util
        end
    end

    %% Frontend / Benchmark Clients Subgraph
    subgraph "Frontend / Benchmark Clients"
        FE("Benchmark Clients"):::frontend
        EA("External Adapters / Integration Layer"):::frontend
    end

    %% Dependencies Subgraph
    subgraph "Dependencies"
        LB("Libraries"):::libs
        SH("Shared Headers"):::libs
    end

    %% Internal Connections in Core Engine
    CR -->|"logs_to"| BM
    BM -->|"optimizes"| BT

    %% Dependencies support Core Engine
    LB -->|"supports"| CS
    SH -->|"defines"| CS

    %% Frontend interacts with Core Engine
    FE -->|"calls"| CS
    EA -->|"integrates"| CS

    %% Support Utilities assist Core Engine
    SY -->|"assists"| CS
    TH -->|"assists"| CS
    UT -->|"supports"| CS

    %% Click Events
    click CS "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore"
    click CR "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/concurrency-recovery"
    click PR "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/profiling"
    click BT "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/storage/btree"
    click BM "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/storage/buffer-manager"
    click SY "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/sync-primitives"
    click TH "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/threads"
    click UT "https://github.com/anchi205/in-mem-store/tree/main/backend/leanstore/utils"
    click FE "https://github.com/anchi205/in-mem-store/tree/main/frontend"
    click EA "https://github.com/anchi205/in-mem-store/tree/main/frontend/shared"
    click LB "https://github.com/anchi205/in-mem-store/tree/main/libs"
    click SH "https://github.com/anchi205/in-mem-store/tree/main/shared-headers"

    %% Styles
    classDef core fill:#fdd,stroke:#333,stroke-width:2px;
    classDef storage fill:#dfd,stroke:#333,stroke-width:2px;
    classDef util fill:#ddf,stroke:#333,stroke-width:2px;
    classDef frontend fill:#ffd,stroke:#333,stroke-width:2px;
    classDef libs fill:#dff,stroke:#333,stroke-width:2px;
```
