# scalar-labs/scalardb architecture

> Architecture diagram of the GitHub repository scalar-labs/scalardb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scalar-labs/scalardb
- Repository: https://github.com/scalar-labs/scalardb

## Overview

##### Project type and purpose
- ScalarDB is a Java library (SDK) providing a universal HTAP engine with ACID transactions and real-time analytics across diverse datastores.
- The repo contains:
– core library (API, common logic, storage implementations, transaction modules)
– two CLI tools (data-loader and schema-loader)
– integration-test harness
– formal TLA+ specifications
– buildSrc plugin, CI/CD workflows

##### Top-level components
- buildSrc (Gradle plugin for JDK configuration)
- core
– com.scalar.db.api (public Java API interfaces: DistributedStorage, Crud, Transaction)
– com.scalar.db.common (abstract base classes, shared utilities)
– com.scalar.db.config (configuration loaders)
– com.scalar.db.service (Guice/Java-DI modules: ProviderManager, StorageModule, TransactionModule)
– com.scalar.db.storage/* (storage provider SPI implementations)
- cassandra, cosmos, dynamo, jdbc, multistorage
– com.scalar.db.transaction/* (transaction engines)
- consensus-commit, singlecrud, JDBC transaction
– com.scalar.db.util (helpers, group-commit utilities)
- data-loader
– cli (DataLoaderCli, command parsers)
– core (ExportManager/ImportManager orchestrating ScalarDB APIs)
- schema-loader
– CLI wrapper for table schema import/alteration via ScalarDB API
- integration-test
– test harness that runs core API tests against actual databases via driver configs
- tla+
– formal specs of consensus-commit protocol

##### Relationships and interactions
- The CLI tools (data-loader, schema-loader) depend on the core library’s API and service modules to perform operations.
- core.api → core.service → core.storage provider → external database (Cassandra/CosmosDB/DynamoDB/RDB)
- core.api → core.service → core.transaction → core.storage → external database
- integration-test invokes core.api against embedded or real test containers of storage providers.
- TLA+ specs describe the consensus-commit transaction protocol; no runtime link, but conceptually sits “beside” core.transaction.
- buildSrc provides build-time Gradle configuration.
- GitHub Actions CI uses build scripts, dockerfile linter, vuln checks, and publishes artifacts.

##### Architectural patterns and design principles
- SPI/Provider pattern: core.api interfaces are implemented by provider modules (storage & transaction).
- Dependency Injection (Service/Module packages) for wiring API → implementations.
- Separation of concerns:
– API layer (com.scalar.db.api)
– Common logic (com.scalar.db.common)
– Storage implementations by backend
– Transaction managers by strategy
– CLI orchestration separated from core logic
- Two-phase commit (consensus-commit) and single-operation transactions provide pluggable transaction strategies.

##### Technologies and frameworks
- Java 8+ (build.gradle, buildSrc plugin)
- Gradle multi-project build
- SPI (java.util.ServiceLoader via META-INF/services)
- External datastores: Cassandra, Azure CosmosDB, AWS DynamoDB, JDBC RDB engines
- CLI tools: picocli or similar (DataLoaderCli)
- Formal methods: TLA+ Toolbox
- CI/CD: GitHub Actions, pre-commit, Dependabot

- Clear labels (e.g., “com.scalar.db.api”, “CassandraProvider”)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Build"
        Bld["buildSrc"]:::tool
    end

    subgraph "CLI Tools" 
        DL_CLI["Data Loader CLI"]:::cli
        DL_Core["Data Loader Core"]:::cli
        SL["schema-loader"]:::cli
    end

    subgraph core
        subgraph "API Layer"
            API["com.scalar.db.api"]:::api
        end
        subgraph "Common/Config"
            Common["com.scalar.db.common"]:::common
            Config["com.scalar.db.config"]:::common
        end
        subgraph "Service Modules"
            Service["com.scalar.db.service"]:::service
        end
        subgraph "Storage Providers"
            Cass["CassandraProvider"]:::storage
            Cosmos["CosmosDBProvider"]:::storage
            Dynamo["DynamoDBProvider"]:::storage
            JDBCStorage["JDBCProvider"]:::storage
            Multi["MultiStorageProvider"]:::storage
        end
        subgraph "Transaction Modules"
            CC["ConsensusCommitTx"]:::tx
            JDBCtx["JDBCTransaction"]:::tx
            Single["SingleCrudTx"]:::tx
        end
        Util["com.scalar.db.util"]:::common
    end

    subgraph "Integration Test"
        IT["integration-test harness"]:::test
    end

    TLA["TLA+ specs"]:::tla

    subgraph "External Datastores"
        DB1[("Cassandra")]:::db
        DB2[("CosmosDB")]:::db
        DB3[("DynamoDB")]:::db
        DB4[("RDBMS")]:::db
    end

    Bld -->|"build-time plugin"| core
    DL_CLI -->|"calls API"| API
    DL_Core -->|"uses API"| API
    SL -->|"uses API"| API
    API -->|"injects"| Service
    Service -->|"loads SPI"| Cass
    Service -->|"loads SPI"| Cosmos
    Service -->|"loads SPI"| Dynamo
    Service -->|"loads SPI"| JDBCStorage
    Service -->|"loads SPI"| Multi
    Service -->|"wires tx modules"| CC
    Service -->|"wires tx modules"| JDBCtx
    Service -->|"wires tx modules"| Single
    CC -->|"coordinates with storage"| Cass
    CC -->|"coordinates with storage"| Cosmos
    CC -->|"coordinates with storage"| Dynamo
    CC -->|"coordinates with storage"| JDBCStorage
    JDBCtx -->|"operates on"| JDBCStorage
    Single -->|"operates on"| Cass
    Single -->|"operates on"| Cosmos
    Single -->|"operates on"| Dynamo
    Single -->|"operates on"| JDBCStorage
    Single -->|"operates on"| Multi
    API -->|"common utilities"| Util
    IT -->|"tests via API"| API
    IT -->|"spins up"| DB1
    IT -->|"spins up"| DB2
    IT -->|"spins up"| DB3
    IT -->|"spins up"| DB4
    CC -.->|"specified by"| TLA

    Cass -->|"data stored in"| DB1
    Cosmos -->|"data stored in"| DB2
    Dynamo -->|"data stored in"| DB3
    JDBCStorage -->|"data stored in"| DB4
    Multi -->|"routes to"| DB1 & DB2 & DB3 & DB4

    classDef api fill:#a2d5f2,stroke:#07689f,color:#07689f;
    classDef common fill:#d3f8e2,stroke:#2d6a4f,color:#2d6a4f;
    classDef service fill:#fff3b0,stroke:#f09,stroke-width:1px,color:#f09;
    classDef storage fill:#ffcad4,stroke:#b5179e,color:#b5179e;
    classDef tx fill:#e2e2e2,stroke:#6c757d,color:#6c757d;
    classDef util fill:#d3f8e2,stroke:#2d6a4f,color:#2d6a4f;
    classDef cli fill:#a0c4ff,stroke:#003566,color:#003566;
    classDef tool fill:#ffc6ff,stroke:#6a0572,color:#6a0572;
    classDef test fill:#ffd6a5,stroke:#ef233c,color:#ef233c;
    classDef tla fill:#e0bbff,stroke:#5f259f,color:#5f259f;
    classDef db fill:#f0efeb,stroke:#adb5bd,color:#495057;

    click Bld "https://github.com/scalar-labs/scalardb/blob/master/buildSrc/src/main/java/com/scalar/db/JdkConfigurationPlugin.java"
    click API "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/api"
    click Common "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/common"
    click Config "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/config"
    click Service "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/service"
    click Cass "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/storage/cassandra"
    click Cosmos "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/storage/cosmos"
    click Dynamo "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/storage/dynamo"
    click JDBCStorage "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/storage/jdbc"
    click Multi "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/storage/multistorage"
    click CC "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/transaction/consensuscommit"
    click JDBCtx "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/transaction/jdbc"
    click Single "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/transaction/singlecrudoperation"
    click Util "https://github.com/scalar-labs/scalardb/tree/master/core/src/main/java/com/scalar/db/util"
    click DL_CLI "https://github.com/scalar-labs/scalardb/blob/master/data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/DataLoaderCli.java"
    click DL_Core "https://github.com/scalar-labs/scalardb/tree/master/data-loader/core/src/main/java/com/scalar/db/dataloader/core"
    click SL "https://github.com/scalar-labs/scalardb/tree/master/schema-loader/src/main/java/com/scalar/db/schemaloader"
    click IT "https://github.com/scalar-labs/scalardb/tree/master/integration-test/src/main/java/com/scalar/db/api"
    click IT "https://github.com/scalar-labs/scalardb/tree/master/integration-test/src/main/java/com/scalar/db/transaction"
    click TLA "https://github.com/scalar-labs/scalardb/blob/master/tla+/consensus-commit/CC.tla"
```
