# jorben/rsync-object-storage architecture

> Architecture diagram of the GitHub repository jorben/rsync-object-storage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jorben/rsync-object-storage
- Repository: https://github.com/jorben/rsync-object-storage

## Overview

##### Main Components
- Local Filesystem: Represents the source directory (and its subdirectories) being monitored.
- File Watcher Module: Implemented in watcher.go, this component listens for file change events (creation, modification, deletion) on the local filesystem.
- Sync Engine:
- Real-Time Sync: Triggered by live file events. The configuration (sync.real_time.enable and hot_delay) controls how frequently changes are batched.
- Scheduled Check/Full Sync: The checkjob.go and associated configuration parameters (start_at and interval) define the periodic full-directory synchronization to reconcile differences between local and remote.
- Configuration Loader: Uses config.yaml and the Go package in the config directory (config.go); it provides runtime settings for local/remote paths, sync modes, ignore rules, and other options.
- Remote Object Storage Client: Connects to S3-like endpoints; this component uses settings from the configuration (e.g., endpoint, secret_id/secret_key) to manage file transfers (uploads and deletions).
- Logging and Error Handling: Centralized via the log, enum, and helper directories. Logging (log.go) delivers console and file output, while enums and helper utilities support error reporting and reusable functions.
- Docker & Deployment Components: Dockerfile and associated GitHub workflow configurations reflect the containerized deployment environment.

##### File Relationships and Interactions
- The main.go acts as the entry point which initially loads configuration and sets up the file watcher.
- File events from watcher.go and file processing from transfer.go route through helper functions (in helper/) and key-value operations (in kv/kv.go) to the Remote Storage Client.
- The configuration (config.yaml) drives the behavior of the sync engine (both real-time and scheduled) and informs logging decisions via log and enum.
- The checkjob.go module appears to handle the scheduled reconciliations of file differences.
- External services (remote object storage endpoint) interact with the sync engine through API calls during file transfers.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Distinct modules for configuration, logging, file monitoring, sync logic, and remote operations.
- Real-Time vs. Scheduled Process Flows: Two distinct synchronization flows (immediate and periodic).
- Modularity: Clearly defined packages (config, enum, helper, kv, log) ensure that each component has a focused responsibility.
- Containerization: Docker and GitHub workflows indicate a modern DevOps approach for continuous integration and deployment.

##### Putting It All Together
- Flow to the center where the "Sync Engine" breaks into two parallel processes (Real-Time and Scheduled Sync).
- Sync Engine to the "Remote Object Storage Client" which then interfaces with the external S3 service.
- Optionally, represent the Docker container ecosystem to indicate deployment and CI/CD pipelines.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Main Application
    MAIN["Entry Point (main.go)"]:::app
    
    %% Local Filesystem and Watcher
    FS["Local Filesystem"]:::source
    WATCHER["File Watcher Module (watcher.go)"]:::watcher

    %% Sync Engines
    RT["Real-Time Sync Engine (transfer.go)"]:::sync
    SCHED["Scheduled Sync Module (checkjob.go)"]:::sync

    %% Remote Storage
    REMOTE["Remote Object Storage Client (storage.go)"]:::remote
    S3(["S3-like Object Storage"]):::external

    %% Configuration Loader Subgraph
    subgraph "Configuration Loader"
        CONFIG1["Config File (config.yaml)"]:::config
        CONFIG2["Config Loader (config.go)"]:::config
    end

    %% Logging and Error Handling Subgraph
    subgraph "Logging & Error Handling"
        LOGGER["Logger (log/log.go)"]:::logging
        CONSTANTS["Constants (enum/constants.go)"]:::logging
        ERRORS["Errors (enum/errors.go)"]:::logging
        HELP["Helper Utils (helper)"]:::logging
    end

    %% Deployment & CI/CD Subgraph
    subgraph "Deployment & CI/CD"
        DOCKER["Dockerfile"]:::deployment
        WORKFLOWS["GitHub Workflows (.github/workflows/)"]:::deployment
    end

    %% Connections
    MAIN -->|"loads config"| CONFIG1
    MAIN -->|"loads config"| CONFIG2
    MAIN -->|"initializes"| WATCHER

    FS -->|"triggers events"| WATCHER
    CONFIG1 -->|"configures"| WATCHER

    WATCHER -->|"triggersSync"| RT

    CONFIG2 -->|"configures"| RT
    CONFIG1 -->|"configures"| SCHED
    CONFIG2 -->|"configures"| SCHED

    RT -->|"uploadsChanges"| REMOTE
    SCHED -->|"fullSync"| REMOTE

    REMOTE -->|"managesTransfer"| S3

    %% Logging from various modules
    MAIN --> LOGGER
    WATCHER --> LOGGER
    RT --> LOGGER
    SCHED --> LOGGER
    REMOTE --> LOGGER

    %% Deployment flow
    WORKFLOWS -->|"triggers build"| DOCKER
    DOCKER -->|"deploys"| MAIN

    %% Click Events
    click WATCHER "https://github.com/jorben/rsync-object-storage/blob/master/watcher.go"
    click RT "https://github.com/jorben/rsync-object-storage/blob/master/transfer.go"
    click SCHED "https://github.com/jorben/rsync-object-storage/blob/master/checkjob.go"
    click CONFIG1 "https://github.com/jorben/rsync-object-storage/blob/master/config.yaml"
    click CONFIG2 "https://github.com/jorben/rsync-object-storage/blob/master/config/config.go"
    click REMOTE "https://github.com/jorben/rsync-object-storage/blob/master/storage.go"
    click LOGGER "https://github.com/jorben/rsync-object-storage/blob/master/log/log.go"
    click CONSTANTS "https://github.com/jorben/rsync-object-storage/blob/master/enum/constants.go"
    click ERRORS "https://github.com/jorben/rsync-object-storage/blob/master/enum/errors.go"
    click HELP "https://github.com/jorben/rsync-object-storage/tree/master/helper/"
    click DOCKER "https://github.com/jorben/rsync-object-storage/tree/master/Dockerfile"
    click WORKFLOWS "https://github.com/jorben/rsync-object-storage/tree/master/.github/workflows/"

    %% Styles
    classDef app fill:#FFEEAD,stroke:#D98C00,stroke-width:2px;
    classDef source fill:#D0E8F2,stroke:#31708F,stroke-width:2px;
    classDef watcher fill:#DFF0D8,stroke:#3C763D,stroke-width:2px;
    classDef sync fill:#FCF8E3,stroke:#8A6D3B,stroke-width:2px;
    classDef remote fill:#D9EDF7,stroke:#31708F,stroke-width:2px;
    classDef external fill:#F2DEDE,stroke:#A94442,stroke-width:2px;
    classDef config fill:#E2F0CB,stroke:#5A8F29,stroke-width:2px;
    classDef logging fill:#F9D6D5,stroke:#C9302C,stroke-width:2px;
    classDef deployment fill:#E6E6FA,stroke:#6A5ACD,stroke-width:2px;
```
