How dolthub/dolt works
1. What this repository is
- This is the main Dolt monorepo: a Git-like versioned SQL database implemented in Go.
- The core product surfaces as both a dolt CLI and a MySQL-compatible server, with the same repository state reachable from either interface.
2. Top-level architectural split
- The repo is organized around three big layers:
- go/store: low-level versioned storage, chunking, hashes, serialization, and remote blob/chunk persistence.
- go/libraries/doltcore: database semantics, repository/environment management, SQL integration, merge/rebase logic, remotes, and server behavior.
- go/cmd/dolt: end-user CLI command parsing and command execution.
- Tests, performance harnesses, Docker assets, workflows, and integration suites surround those layers.
3. Core storage engine
- go/store is the persistence foundation. It contains:
- chunk stores and content-addressed storage (chunks, nbs, blobstore),
- value/type systems (types, val, prolly),
- commit and dataset primitives (datas, hash, spec),
- archival and compaction utilities.
- This layer is the boundary between logical database objects and physical storage. It is where objects become hashes/chunks and where local, file, S3-like, Git-like, OCI, and other persistence backends are abstracted.
4. Versioned database model
- Dolt models data as a repository with branches, tags, commits, working sets, stashes, and refs.
- In go/libraries/doltcore/doltdb, env, ref, merge, rebase, cherry_pick, and revert, the repo logic builds on go/store to implement Git-style operations on tables rather than files.
- The key conceptual flow is: working set changes -> staged state -> commit objects -> branch refs -> merge/rebase history.
5. SQL/database layer
- go/libraries/doltcore/sqle adapts the versioned store into a SQL engine and MySQL-compatible database implementation.
- This layer includes:
- database/session/transaction management,
- table adapters and writers,
- system tables, table functions, and stored procedures for version-control operations,
- schema evolution, constraints, foreign keys, and query execution helpers.
- The README’s dolt_* SQL procedures and tables are implemented here: they are the SQL façade over the version-control engine.
6. CLI and server entrypoints
- go/cmd/dolt is the command-line front door. It defines the command tree for init, add, commit, branch, merge, sql, sql-server, push/pull, diff, status, etc.
- go/libraries/doltcore/sqlserver and go/cmd/dolt/commands/sqlserver wire the server runtime, configs, auth, metrics, and MySQL protocol behavior.
- The CLI and server both operate on the same repo/environment abstraction, so they are different interfaces into one underlying database model.
7. Repository/environment abstraction
- go/libraries/doltcore/env is the runtime glue that binds a local directory, repo state, remotes, credentials, and the active branch/workspace together.
- This is the main boundary between “filesystem/project directory” and “Dolt repository.”
- Many user-facing commands and server operations go through this layer to locate state, resolve refs, handle multiple repos, and manage config and remotes.
8. Remote and replication architecture
- Remote behavior is split across dbfactory, remotesrv, remotestorage, grpcendpoint, and related proto definitions under go/gen/proto and proto/.
- Supports several transport and storage modes: HTTP, gRPC, file, Git, cloud object storage, SSH, and provider-specific backends.
- Replication and remote sync are first-class paths, not just “git push/pull equivalents”; the repo includes dedicated remotes service APIs and replication APIs.
9. Data import/export and file-format boundaries
- go/libraries/doltcore/table and mvdata handle moving data between external formats and Dolt tables.
- Supported formats include CSV, JSON, Parquet, XLSX, and SQL-oriented exports, with typed and untyped readers/writers.
- This is the boundary where external datasets enter the versioned table model and where table contents can be exported back out.
10. Schema, diff, and merge mechanics
- go/libraries/doltcore/schema, row, rowconv, diff, merge, index, fk, and constraint implement relational behavior on top of the versioned store.
- This includes primary/secondary indexes, keyless rows, foreign keys, constraint checks, and schema diffs/merges.
- Merge conflict handling and table-level diffs are central to the product and appear both in CLI commands and SQL system tables.
11. Operational/server features
- go/libraries/doltcore/sqle/cluster, clusterdb, binlogreplication, statspro, auto_gc, and read_replica_database support server-side operational features.
- These cover binlog compatibility/replication, statistics collection, background GC, cluster state, read replicas, and server hooks.
- The repo is not just a local database; it has runtime features for running Dolt as a managed service or in clustered/replicated setups.
12. Tooling, testing, and release pipeline
- The .github/workflows directory is very large and encodes CI/CD, compatibility tests, SQL/server integration, benchmarks, fuzzing, release publishing, Docker image builds, and email/reporting automation.
- integration-tests, go/performance, and go/store/*_test.go provide extensive coverage from storage to SQL compatibility to client integration.
- This implies the architecture is validated at multiple levels: storage, library, CLI, server, remote sync, and client protocol.
13. Infrastructure and packaging
- docker/, release workflows, and utility packages under go/utils support packaging and deployment.
- The README’s installation paths reflect this: binaries, Docker images, package managers, and source builds all target the same Go codebase.
14. How to draw the architecture diagram
- Place go/store at the bottom as the durable content-addressed storage and object model.
- Above it, place doltcore as the domain/runtime layer: repo state, SQL engine, merge/rebase, remotes, schema, and server behaviors.
- At the top, show two primary entrypoints: the dolt CLI and the MySQL-compatible SQL server.
- Draw side boundaries for external systems: remote storage/providers, SQL clients, file import/export formats, and CI/test infrastructure.
Main entry points: sqle SQL Engine · sqlserver SQL Server · store Store API · doltdb Repo Model
Components 18
Entry points
User-facing command and server interfaces.
- Dolt CLI · End-user command front door for repo and SQL actions.
go/cmd/dolt - SQL Server · MySQL protocol server exposing the same repository state.
go/libraries/doltcore/sqlserver
Runtime core
Repository state, SQL engine, schema, merge, and environment logic.
- SQL Engine · Adapts Dolt repositories into SQL sessions, transactions, and table access.
go/libraries/doltcore/sqle - Env Layer · Binds local filesystem state, refs, remotes, credentials, and active workspace.
go/libraries/doltcore/env - Repo Model · Repository objects, commits, working sets, refs, tags, and history operations.
go/libraries/doltcore/doltdb - Merge Ops · Merge, rebase, cherry-pick, revert, and related history-editing logic.
go/libraries/doltcore/merge - Schema Model · Schemas, columns, indexes, constraints, foreign keys, and type compatibility.
go/libraries/doltcore/schema - Row Data · Keyed and keyless row representations plus conversion helpers.
go/libraries/doltcore/row - Row Conv · Transforms between table schemas, SQL rows, and Dolt row formats.
go/libraries/doltcore/rowconv - Table I/O · Moves data between external file formats and Dolt tables.
go/libraries/doltcore/table - Remote Sync · Push, pull, replication, and remote storage transport behavior.
go/libraries/doltcore/remotestorage - Remote Server · HTTP/gRPC server for chunk and replication APIs.
go/libraries/doltcore/remotesrv - DB Factory · Creates local and remote-backed repositories for file, cloud, git, SSH, OCI, and gRPC endpoints.
go/libraries/doltcore/dbfactory
Storage core
Low-level content-addressed storage and value encoding.
- Store API · Chunk stores, blobs, manifests, archives, and persistence backends.
go/store - Prolly Trees · Tree structures, diffs, merges, and mutable maps used for versioned table storage.
go/store/prolly - Types & Values · Primitive types, tuples, refs, codecs, and value storage primitives.
go/store/types
External interfaces
Remote systems, file formats, and test/integration surfaces.
- Tests & CI · Integration suites, performance harnesses, and GitHub workflow automation.
.github/workflows - Clients & Formats · SQL clients, dumps, CSV/JSON/Parquet/XLSX, Docker packaging, and deployment surfaces.
integration-tests/mysql-client-tests
Connections 23
- Dolt CLI → Env Layer: opens repo · CLI commands locate and operate on repository/environment state.
- Dolt CLI → SQL Engine: runs SQL · The CLI shells into the embedded SQL engine and procedures.
- SQL Server → SQL Engine: hosts · The server runtime exposes the SQL engine over MySQL protocol.
- SQL Server → Env Layer: binds env · Server startup resolves repo state, config, auth, and remotes.
- SQL Engine → Repo Model: reads/writes · SQL sessions and procedures manipulate the underlying versioned repository model.
- SQL Engine → Schema Model: enforces schema · DDL, constraints, foreign keys, and table metadata are handled here.
- SQL Engine → Row Data: maps rows · SQL execution converts between engine rows and Dolt row encodings.
- Row Conv → Row Data: converts · Row conversion bridges schema-aware SQL values and storage rows.
- Table I/O → Row Conv: normalizes · Import/export pipelines translate external records into table rows.
- Table I/O → Clients & Formats: imports/exports · CSV, JSON, Parquet, XLSX, and SQL-oriented dumps cross this boundary.
- Repo Model → Store API: persists · Repository objects are stored as chunks, manifests, and content-addressed data.
- Repo Model → Merge Ops: history ops · Commit and working-set state feed merge, rebase, cherry-pick, and revert flows.
- Merge Ops → Prolly Trees: diffs/merges · Merge mechanics rely on prolly-tree traversal and mutable map operations.
- Schema Model → Types & Values: encodes · Schemas and constraints rely on typed values, tuples, and codecs.
- Store API → Types & Values: stores values · The storage layer persists typed values and refs as content-addressed objects.
- Store API → Prolly Trees: backs trees · Chunk stores and persistence backends support prolly-tree storage and indexing.
- Remote Sync → Store API: syncs chunks · Remote sync moves chunks and repository data across storage backends.
- Remote Server → Remote Sync: serves · The remotes service exposes sync and replication APIs over network protocols.
- DB Factory → Remote Sync: constructs remotes · Factories assemble local, cloud, git, SSH, OCI, file, and gRPC-backed repos.
- Tests & CI → Dolt CLI: validates · CLI behavior is exercised by integration and workflow automation.
- Tests & CI → SQL Server: validates · Server compatibility, replication, and protocol behavior are covered by tests.
- Clients & Formats → SQL Server: connects to · External SQL clients and ORMs talk to the MySQL-compatible server.
- Clients & Formats → Table I/O: feeds data · External files and dumps are imported/exported through table I/O pipelines.