How altinity/clickhouse-backup works
This project is a command-line utility written in Go for managing ClickHouse backups. It’s designed to run on the same host (or in the same container/pod) as the ClickHouse server because it establishes file system hard links to ClickHouse data directories and then handles upload/download operations for backup files. In addition, it can run an API server for asynchronous operations.
Main Components
• CLI / Command Executor: Represent the entry point (located in the “cmd/clickhouse-backup” directory) that processes backup–related commands (create, restore, upload, download, etc.).
• Backup Engine (pkg/backup): This module handles core backup logic, including freezing tables (via ALTER TABLE … FREEZE), file system hard linking, and restoration. It interacts directly with file system storage.
• ClickHouse Connector (pkg/clickhouse): This module facilitates communication with the running ClickHouse server. It issues queries (both for backup and restore) and includes versioning and utility functions.
• Configuration Module (pkg/config): Reads and manages configurations (including overriding defaults via environment variables and command-line parameters).
• Storage Abstraction (pkg/storage): Contains implementations for multiple remote storage providers (such as S3, GCS, Azure Blob, FTP, SFTP, COS, and the custom method). It is responsible for splitting, compressing (using different formats), and transferring backup data.
• API Server (pkg/server): Exposes a REST API that mirrors the CLI functions. It provides endpoints for backup creation, restoration, upload/download, status monitoring, and administrative operations.
• Auxiliary Modules: These include the “pkg/common” for shared types, “pkg/custom” for custom storage methods, “pkg/filesystemhelper” for interacting with the file system, “pkg/keeper” for coordination logic, “pkg/metadata” for backup metadata, “pkg/partition” for partition handling, “pkg/resumable” for stateful operations, “pkg/status” for reporting, and “pkg/utils” for utility functions.
• Testing Modules: The test directories (both integration tests under “test/integration” and testflows in “test/testflows”) provide coverage across backup/restore operations and remote storage integration.
Relationships and Interactions
• The CLI or API calls enter the system via the “cmd/clickhouse-backup” entry point.
• Configuration values are loaded early on (using the pkg/config module), influencing behavior in subsequent backups/restores.
• The Backup Engine (pkg/backup) interacts with the ClickHouse Connector (pkg/clickhouse) to issue freezing commands. It then utilizes file system helpers to create hard links, effectively “capturing” the state of ClickHouse data.
• Once a backup is created locally, the Storage Abstraction (pkg/storage) takes over to package and upload the backup to a remote storage provider. The storage modules can dynamically select an implementation based on the current config (e.g., s3, gcs, custom, etc.). The engine supports concurrent uploads and downloads, as well as resumable state (handled by pkg/resumable).
• Restoration works in a reverse direction: the backup data is downloaded (or restored from a local state) and then passed to the ClickHouse Connector for executing “ATTACH PART” queries after moving files into a “detached” folder.
• The API Server (pkg/server) allows operations to be run asynchronously, and it also provides additional monitoring endpoints (e.g., for backup status, listing backups, etc.). There’s coordination between the API and the backup logic to support long-running operations.
• External deployment and automation concerns (as seen with Dockerfile, Vagrantfile, and GitHub workflows) are separate from the core runtime but should be noted in the deployment layer.
Architectural Patterns and Design Principles
• Modular Design: The project is split into multiple “pkg” submodules that each focus on a single domain – for example, backup logic, storage, connectivity, configuration – thus promoting separation of concerns.
• Command Pattern: The CLI is organized to allow different commands (create, restore, upload, download, watch, etc.) which isolated business logic through the respective modules.
• Abstraction for Storage: Using interfaces to encapsulate differing remote storage providers allows the backup engine to be extensible to support cloud storage as well as custom remote commands.
• Concurrency & Parallelism: The design explicitly supports controlling concurrency (upload_concurrency, download_concurrency, etc.) to efficiently handle multi-disk backups, multi-part uploads, and resumable operations.
• API-driven Operations: By exposing a RESTful API (via pkg/server), the design supports asynchronous operations and integration with external monitoring and orchestration tools.