How kuberlab/pluk works
Main Components
• Pluk Server: This is the core backend process (pluksrv.go) responsible for handling HTTP requests, managing data chunks, file metadata, and communicating with the DB.
• API Layer: Under pkg/api, you have RESTful API endpoints (with functions for handling datasets, files, and versioning) that connect front-end CLI clients and possibly internal FUSE connections.
• Database: Under pkg/db and pkg/db/gorm, all interactions with the chosen database (sqlite3 by default, but also mysql/postgres are supported) are defined. This database stores the virtual filesystem’s metadata, dataset versions, and file chunks info.
• FUSE Filesystem: The plukefs binary (and its supporting code in pkg/fuse) allows mounting datasets as read-only filesystems. It provides an interface between the kernel’s filesystem calls and the underlying dataset management logic.
• CLI Tool – kdataset: Located in the cmd/kdataset folder, this tool offers a command-line interface for operations such as push, pull, list, version management, and deletion.
• Auxiliary Tools: Additional commands such as delete_wrong_chunks and update_chunks handle specific maintenance or synchronization tasks.
• gRPC Communication: The pkg/grpc package indicates that there is support for gRPC-based communication, perhaps for internal communication (such as master-slave sync where masters are defined via the environment variable).
• External Configuration & Environment: Environment variables (like DATA_DIR, DB_NAME, AUTH_VALIDATION, and MASTERS) play a crucial role in configuring the system and its interactions with external services (e.g., an authentication proxy).
Relationships and Interactions
• The Pluk Server (backend) exposes a REST API (and possibly WebSocket endpoints) to handle dataset operations.
• Requests may come from both external clients (through the kdataset CLI) and internally from the FUSE filesystem (plukefs) when mounting a dataset.
• The server communicates with the Database layer to read/write dataset metadata, chunk data references, version info, etc.
• gRPC communications are utilized for internal master-slave replication/communication where slave instances interact with a master (using environment-specified internal keys).
• The CLI tool (kdataset) interacts with the Pluk Server via HTTP (or through a configuration provided pluk_url) to perform its operations.
• Docker containers (referenced by Dockerfile and Dockerfile.plukefs) encapsulate the necessary environment, ensuring correct volumes for DATA_DIR and DB access.
• Maintenance utilities (delete_wrong_chunks, update_chunks) likely have shorter, specialized data flows targeting cleanup or synchronization tasks.
Important Architectural Patterns and Design Principles
• Modular design: Different commands and functionalities are separated into distinct binaries (pluk, plukefs, kdataset) and packages (pkg/api, pkg/db, pkg/fuse, etc.), reflecting a separation of concerns.
• Configurability and environment-based behavior: Use of environment variables to configure the storage (DATA_DIR), DB type/parameters, and authentication tools.
• Client–server model: With a clearly defined API layer that decouples client tools (CLI, FUSE) from the core data management logic.
• Containerization: Support and guidance for running the system inside Docker, ensuring portability.
• Extensibility: Through clear command-line interfaces and configuration files (YAML for kdataset), it is designed to be integrated and extended as needed.