How jorben/rsync-object-storage works
Main Components
• Local Filesystem: Represents the source directory (and its subdirectories) being monitored.
• File Watcher Module: Implemented in watcher.go, this component listens for file change events (creation, modification, deletion) on the local filesystem.
• Sync Engine:
• Real-Time Sync: Triggered by live file events. The configuration (sync.real_time.enable and hot_delay) controls how frequently changes are batched.
• Scheduled Check/Full Sync: The checkjob.go and associated configuration parameters (start_at and interval) define the periodic full-directory synchronization to reconcile differences between local and remote.
• Configuration Loader: Uses config.yaml and the Go package in the config directory (config.go); it provides runtime settings for local/remote paths, sync modes, ignore rules, and other options.
• Remote Object Storage Client: Connects to S3-like endpoints; this component uses settings from the configuration (e.g., endpoint, secret_id/secret_key) to manage file transfers (uploads and deletions).
• Logging and Error Handling: Centralized via the log, enum, and helper directories. Logging (log.go) delivers console and file output, while enums and helper utilities support error reporting and reusable functions.
• Docker & Deployment Components: Dockerfile and associated GitHub workflow configurations reflect the containerized deployment environment.
File Relationships and Interactions
• The main.go acts as the entry point which initially loads configuration and sets up the file watcher.
• File events from watcher.go and file processing from transfer.go route through helper functions (in helper/) and key-value operations (in kv/kv.go) to the Remote Storage Client.
• The configuration (config.yaml) drives the behavior of the sync engine (both real-time and scheduled) and informs logging decisions via log and enum.
• The checkjob.go module appears to handle the scheduled reconciliations of file differences.
• External services (remote object storage endpoint) interact with the sync engine through API calls during file transfers.
Architectural Patterns and Design Principles
• Separation of Concerns: Distinct modules for configuration, logging, file monitoring, sync logic, and remote operations.
• Real-Time vs. Scheduled Process Flows: Two distinct synchronization flows (immediate and periodic).
• Modularity: Clearly defined packages (config, enum, helper, kv, log) ensure that each component has a focused responsibility.
• Containerization: Docker and GitHub workflows indicate a modern DevOps approach for continuous integration and deployment.
Putting It All Together
• Flow to the center where the "Sync Engine" breaks into two parallel processes (Real-Time and Scheduled Sync).
• Sync Engine to the "Remote Object Storage Client" which then interfaces with the external S3 service.
• Optionally, represent the Docker container ecosystem to indicate deployment and CI/CD pipelines.