How scalar-labs/scalardb works
Project type and purpose
• ScalarDB is a Java library (SDK) providing a universal HTAP engine with ACID transactions and real-time analytics across diverse datastores.
• The repo contains:
– core library (API, common logic, storage implementations, transaction modules)
– two CLI tools (data-loader and schema-loader)
– integration-test harness
– formal TLA+ specifications
– buildSrc plugin, CI/CD workflows
Top-level components
• buildSrc (Gradle plugin for JDK configuration)
• core
– com.scalar.db.api (public Java API interfaces: DistributedStorage, Crud, Transaction)
– com.scalar.db.common (abstract base classes, shared utilities)
– com.scalar.db.config (configuration loaders)
– com.scalar.db.service (Guice/Java-DI modules: ProviderManager, StorageModule, TransactionModule)
– com.scalar.db.storage/* (storage provider SPI implementations)
• cassandra, cosmos, dynamo, jdbc, multistorage
– com.scalar.db.transaction/* (transaction engines)
• consensus-commit, singlecrud, JDBC transaction
– com.scalar.db.util (helpers, group-commit utilities)
• data-loader
– cli (DataLoaderCli, command parsers)
– core (ExportManager/ImportManager orchestrating ScalarDB APIs)
• schema-loader
– CLI wrapper for table schema import/alteration via ScalarDB API
• integration-test
– test harness that runs core API tests against actual databases via driver configs
• tla+
– formal specs of consensus-commit protocol
Relationships and interactions
• The CLI tools (data-loader, schema-loader) depend on the core library’s API and service modules to perform operations.
• core.api → core.service → core.storage provider → external database (Cassandra/CosmosDB/DynamoDB/RDB)
• core.api → core.service → core.transaction → core.storage → external database
• integration-test invokes core.api against embedded or real test containers of storage providers.
• TLA+ specs describe the consensus-commit transaction protocol; no runtime link, but conceptually sits “beside” core.transaction.
• buildSrc provides build-time Gradle configuration.
• GitHub Actions CI uses build scripts, dockerfile linter, vuln checks, and publishes artifacts.
Architectural patterns and design principles
• SPI/Provider pattern: core.api interfaces are implemented by provider modules (storage & transaction).
• Dependency Injection (Service/Module packages) for wiring API → implementations.
• Separation of concerns:
– API layer (com.scalar.db.api)
– Common logic (com.scalar.db.common)
– Storage implementations by backend
– Transaction managers by strategy
– CLI orchestration separated from core logic
• Two-phase commit (consensus-commit) and single-operation transactions provide pluggable transaction strategies.
Technologies and frameworks
• Java 8+ (build.gradle, buildSrc plugin)
• Gradle multi-project build
• SPI (java.util.ServiceLoader via META-INF/services)
• External datastores: Cassandra, Azure CosmosDB, AWS DynamoDB, JDBC RDB engines
• CLI tools: picocli or similar (DataLoaderCli)
• Formal methods: TLA+ Toolbox
• CI/CD: GitHub Actions, pre-commit, Dependabot
• Clear labels (e.g., “com.scalar.db.api”, “CassandraProvider”)