# simerplaha/swaydb architecture

> Architecture diagram of the GitHub repository simerplaha/swaydb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/simerplaha/swaydb
- Repository: https://github.com/simerplaha/swaydb

## Overview

##### Project type and purpose
- This is SwayDB, a modular, high-performance persistent and in-memory key-value storage engine for the JVM. It is not a web service but a library composed of multiple SBT modules that provide APIs (Map, Set, Queue, MultiMap, Transactions, Streams, TTL, etc.), a non-blocking core engine, pluggable persistence layers, compression, caching and extension points for functional and reactive use-cases.

##### File structure
- Top-level modules (visible in the root):
- actor: orchestration, scheduling, asynchronous hooks
- configs: default and builder configurations
- core-*: building blocks of the engine (cache, compression, interop, queue, skiplist, series, performance tests, tools)
- core: main engine entry point (CoreInitializer, CoreState, Compactor, Segment manager, File sweeper)
- effect: IO abstraction layer
- macros: codegen utilities
- serializers, slice, stream: serialization, slice data structure, streaming APIs
- stress, core-stress: stress testing harnesses
- utils: common utilities
- x-interop-*: adapters for Boopickle, Cats-Effect, Monix, ZIO
- swaydb, swaydb-java: user-facing Scala & Java APIs
- testkit, docs, project, build.sbt, version.sbt, .github, LICENSE, README.md

- Patterns:
- Layered architecture: API layer (swaydb/, swaydb-java/) → Effect/interop → Core engine (core/, core-*) → Low-level modules (file I/O, segment, block, cache, compression, skiplist, queue, series) → OS/disk
- Actor-based scheduling and thread pools for non-blocking operations
- SBT multi-project build orchestrating module dependencies

##### README for additional insights
- Highlights:
- “non-blocking core” with configurable blocking/reactive APIs
- Single/multi-disk persistence, in-memory, or eventually persistent modes
- Streams, TTL, Range operations, Configurable compression (LZ4 & Snappy)
- Extensibility: build new data structures, custom serialisers, builders

##### Details to capture
– Thread pools: IO, compaction, scheduler
– Cache layers vs on-disk segments vs logs
– Compaction workflow: assignment → throttle → merge → write
– Extensibility points: new data structures, custom serializers, custom config

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% API Layer
    subgraph "API Layer"
        ScalaAPI["Scala API\nswaydb"]:::api
        JavaAPI["Java API\nswaydb-java"]:::api
    end

    %% Configuration & Builder Layer
    subgraph "Configuration Layer"
        Configs["Configs"]:::config
        ConfigWizard["ConfigWizard & Builders"]:::config
    end

    %% Effect & Interop Layer
    subgraph "Effect & Interop Layer"
        Effect["Effect & IO"]:::effect
        Boopickle["Boopickle Adapter"]:::ext
        CatsEffect["Cats-Effect Adapter"]:::ext
        Monix["Monix Adapter"]:::ext
        ZIO["ZIO Adapter"]:::ext
    end

    %% Actor & Scheduler Layer
    subgraph "Actor & Scheduler Layer"
        Actor["Actor & Scheduler"]:::effect
    end

    %% Core Engine
    subgraph "Core Engine"
        CoreInit["CoreInitializer & CoreState"]:::core
        Compactor["Compactor & Tasks"]:::core
        Sweeper["File Sweeper"]:::core
    end

    %% Storage Subsystems
    subgraph "Storage Subsystems"
        FileIO["File I/O"]:::storage
        Segment["Segment Management"]:::storage
        Block["Block & Compression"]:::storage
        DataStruct["In-Memory Data Structures"]:::storage
        Cache["Caching Layer"]:::storage
    end

    %% Utilities & Tools
    subgraph "Utilities & Tools"
        Macros["Macros"]:::ext
        Serializers["Serialization"]:::ext
        Utils["Utils"]:::ext
        Repair["AppendixRepairer"]:::ext
        Stream["Streaming API"]:::ext
    end

    %% External Disks
    Disk1["Disk"]:::storage
    Disk2["Disk"]:::storage

    %% Connections
    ScalaAPI -->|"loads configs"| Configs
    ScalaAPI -->|"executes ops"| Effect
    JavaAPI -->|"loads configs"| Configs
    JavaAPI -->|"executes ops"| Effect
    Configs -->|"provides settings"| ConfigWizard
    ConfigWizard -->|"builds IO models"| Effect
    Boopickle -->|"serializes data"| Effect
    CatsEffect -->|"provides IO"| Effect
    Monix -->|"provides IO"| Effect
    ZIO -->|"provides IO"| Effect
    Effect -->|"submits tasks"| Actor
    Actor -->|"enqueues tasks"| CoreInit
    CoreInit -->|"dispatches to"| FileIO
    CoreInit -->|"manages"| Segment
    CoreInit -->|"uses cache"| Cache
    CoreInit -->|"uses compression"| Block
    CoreInit -->|"uses data structures"| DataStruct
    CoreInit -->|"schedules cleanup"| Sweeper
    Compactor -->|"merges segments"| Segment
    Compactor -->|"writes data"| FileIO
    Sweeper -->|"deletes old files"| FileIO
    Block -->|"reads/writes blocks"| FileIO
    DataStruct -->|"caches entries"| Cache
    Cache -->|"persists to"| FileIO
    FileIO --> Disk1
    FileIO --> Disk2
    Compactor -.->|"throttle feedback"| CoreInit

    %% Click Events
    click ScalaAPI "https://github.com/simerplaha/swaydb/tree/master/swaydb/src/main/scala/swaydb"
    click JavaAPI "https://github.com/simerplaha/swaydb/tree/master/swaydb-java/src/main/scala/swaydb/java"
    click Configs "https://github.com/simerplaha/swaydb/tree/master/configs/src/main/scala/swaydb/configs"
    click ConfigWizard "https://github.com/simerplaha/swaydb/blob/master/core-interop/src/main/scala/swaydb/config/ConfigWizard.scala"
    click ConfigWizard "https://github.com/simerplaha/swaydb/tree/master/core-interop/src/main/scala/swaydb/config/builder"
    click Effect "https://github.com/simerplaha/swaydb/tree/master/effect/src/main/scala/swaydb/effect"
    click Actor "https://github.com/simerplaha/swaydb/tree/master/actor/src/main/scala/swaydb"
    click CoreInit "https://github.com/simerplaha/swaydb/blob/master/core/src/main/scala/swaydb/core/CoreInitializer.scala"
    click CoreInit "https://github.com/simerplaha/swaydb/blob/master/core/src/main/scala/swaydb/core/CoreState.scala"
    click Compactor "https://github.com/simerplaha/swaydb/blob/master/core/src/main/scala/swaydb/core/compaction/Compactor.scala"
    click Compactor "https://github.com/simerplaha/swaydb/tree/master/core/src/main/scala/swaydb/core/compaction/task"
    click Sweeper "https://github.com/simerplaha/swaydb/blob/master/core/src/main/scala/swaydb/core/file/sweeper/FileSweeper.scala"
    click FileIO "https://github.com/simerplaha/swaydb/tree/master/core/src/main/scala/swaydb/core/file"
    click Segment "https://github.com/simerplaha/swaydb/tree/master/core/src/main/scala/swaydb/core/segment"
    click Block "https://github.com/simerplaha/swaydb/tree/master/core/src/main/scala/swaydb/core/segment/block"
    click Block "https://github.com/simerplaha/swaydb/tree/master/core-compression/src/main/scala/swaydb/core/compression"
    click DataStruct "https://github.com/simerplaha/swaydb/tree/master/core-skiplist/src/main/scala/swaydb/core/skiplist"
    click DataStruct "https://github.com/simerplaha/swaydb/tree/master/core-series/src/main/scala/swaydb/core/series"
    click DataStruct "https://github.com/simerplaha/swaydb/tree/master/core-queue/src/main/scala/swaydb/core/queue"
    click Cache "https://github.com/simerplaha/swaydb/tree/master/core-cache/src/main/scala/swaydb/core/cache"
    click Macros "https://github.com/simerplaha/swaydb/tree/master/macros/src/main/scala/swaydb/macros"
    click Serializers "https://github.com/simerplaha/swaydb/tree/master/serializers/src/main/scala/swaydb/serializers"
    click Utils "https://github.com/simerplaha/swaydb/tree/master/utils/src/main/scala/swaydb/utils"
    click Repair "https://github.com/simerplaha/swaydb/blob/master/core-tools/src/main/scala/swaydb/core/tool/AppendixRepairer.scala"
    click Stream "https://github.com/simerplaha/swaydb/tree/master/stream/src/main/scala/swaydb/stream"
    click Stream "https://github.com/simerplaha/swaydb/blob/master/stream/src/main/scala/swaydb/Streamer.scala"
    click Boopickle "https://github.com/simerplaha/swaydb/blob/master/x-interop-boopickle/src/main/scala/swaydb/serializers/BooPickle.scala"
    click CatsEffect "https://github.com/simerplaha/swaydb/blob/master/x-interop-cats-effect/src/main/scala/swaydb/cats/effect/Bag.scala"
    click Monix "https://github.com/simerplaha/swaydb/blob/master/x-interop-monix/src/main/scala/swaydb/monix/Bag.scala"
    click ZIO "https://github.com/simerplaha/swaydb/blob/master/x-interop-zio/src/main/scala/swaydb/zio/Bag.scala"

    %% Styles
    classDef api fill:#cce5ff,stroke:#004085,color:#004085;
    classDef config fill:#d4edda,stroke:#155724,color:#155724;
    classDef effect fill:#ffeeba,stroke:#856404,color:#856404;
    classDef core fill:#f8d7da,stroke:#721c24,color:#721c24;
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef ext stroke-dasharray: 5 5,fill:none,stroke:#17a2b8,color:#17a2b8;
```
