# chandras/data-sync architecture

> Architecture diagram of the GitHub repository chandras/data-sync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chandras/data-sync
- Repository: https://github.com/chandras/data-sync

## Overview

##### Project type and purpose
- This is an infrastructure‐as‐code project that provisions and deploys a “high‐capacity data migration” tool into a Kubernetes cluster via Terraform and Helm.
- Its goal is to automate large‐scale object‐copy operations in cloud storage (e.g., migrate or sync millions of objects) using Argo Workflows, and to provide monitoring via VictoriaMetrics + Grafana.

##### File structure
- Top level holds Terraform configuration (.tf, .tfvars) and a nested folder named `high-capacity-data-migration-tool-main`.
- Within that root module:
- argo-workflow-template.tf + argo-workflow/…yaml → Terraform resources and raw YAML workflows for Argo.
- helm.tf + helm/*.yaml → Helm chart definitions for Argo, Grafana.
- ingress.tf → Ingress (likely NGINX) to expose services.
- lke.tf, provider.tf → Kubernetes provider configuration (Linode Kubernetes Engine or similar).
- victoriametrics.tf + victoriametrics/*.yaml → Deploy VictoriaMetrics (VMAgent, VMPodScrape, VMSingle) for metrics.
- configmaps.tf, secrets.tf → ConfigMaps/Secrets for passing credentials and config.
- variables.tf, locals.tf, output.tf → Parameterization of the Terraform module.

##### README
- README is minimal (“# data-sync”). It implies the core functionality: data synchronization between object stores via Argo workflows.

##### Main components & interactions
a. Components:
1. Terraform (IaC orchestrator)
2. Kubernetes Cluster (LKE)
3. Ingress Controller (exposes HTTP endpoints)
4. Argo Workflows Server (control plane)
5. Argo Workflow Pods (worker tasks for data transfer)
6. Source Object Storage (e.g., S3, GCS)
7. Destination Object Storage
8. VictoriaMetrics (VMSingle for TSDB, VMAgent + PodScrape)
9. Grafana Dashboard
b. Relationships & data flows:
- Terraform → provisions → Kubernetes resources (Argo, Ingress, Monitoring, ConfigMaps, Secrets).
- User/API → submits migration jobs → Argo Workflows Server → schedules pods.
- Argo worker pods → read/write → Source & Dest Storage.
- VMAgent & PodScrape → collect metrics from Argo pods & cluster → VMSingle.
- Grafana → queries VMSingle → displays dashboards.
- Ingress → routes external traffic → Argo UI & Grafana UI.
c. Architectural patterns & principles:
- Declarative infrastructure (Terraform + Helm).
- Control plane / worker pods separation (Argo).
- Observability via sidecar or agent pattern (VictoriaMetrics).
- Config separation (ConfigMaps & Secrets).
d. Key technologies & frameworks:
- Terraform, Helm, Kubernetes (LKE).
- Argo Workflows.
- VictoriaMetrics.
- Grafana.
- Cloud object storage (S3/GCS).

##### Final recommendation
Layer 1: Provisioning
Layer 2: Control & Data Plane
Layer 3: External Interfaces & Monitoring
- Keep it detailed: show ConfigMaps/Secrets as side components feeding Argo & workers.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Provisioning Layer
    subgraph "Provisioning Layer"
        Terraform["Terraform IaC Orchestrator"]:::infra
    end

    %% Kubernetes Cluster Layer
    subgraph "Kubernetes Cluster (LKE)" 
        Cluster["LKE Kubernetes Cluster"]:::infra
        Ingress["Ingress Controller (NGINX)"]:::infra
        subgraph "Argo Control Plane"
            ArgoServer["Argo Workflows Server"]:::control
            ArgoTemplates["Argo Workflow Templates"]:::control
        end
        subgraph "Data Plane"
            Worker["Argo Workflow Pods (Workers)"]:::data
            SourceStorage["Source Object Storage"]:::data
            DestStorage["Destination Object Storage"]:::data
        end
        subgraph "Config & Secrets"
            ConfigMaps["ConfigMaps"]:::infra
            Secrets["Secrets"]:::infra
        end
        subgraph "Monitoring Stack"
            VMAgent["VMAgent"]:::monitor
            VMPodScrape["VMPodScrape"]:::monitor
            VMSingle["VMSingle (TSDB)"]:::monitor
            Grafana["Grafana Dashboard"]:::monitor
        end
    end

    %% External Entities
    User["User / API Client"]:::external

    %% Relationships
    User -->|"submits jobs"| ArgoServer
    Terraform -->|"provisions"| Cluster
    Cluster --> Ingress
    Ingress -->|"routes HTTP"| ArgoServer
    Ingress -->|"routes HTTP"| Grafana
    ArgoServer -->|"schedules"| Worker
    Worker -->|"reads/writes Object API"| SourceStorage
    Worker -->|"reads/writes Object API"| DestStorage
    Worker -->|"emits metrics"| VMAgent
    VMAgent -->|"scrapes metrics"| VMPodScrape
    VMPodScrape -->|"stores metrics"| VMSingle
    Grafana -->|"queries metrics"| VMSingle
    ConfigMaps --> ArgoServer
    ConfigMaps --> Worker
    Secrets --> ArgoServer
    Secrets --> Worker

    %% Click Events
    click Terraform "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/provider.tf"
    click Terraform "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/variables.tf"
    click Terraform "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/locals.tf"
    click Terraform "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/output.tf"
    click Terraform "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/data.tf"
    click Terraform "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/config.tfvars"
    click Cluster "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/lke.tf"
    click Ingress "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/ingress.tf"
    click ArgoServer "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/helm.tf"
    click ArgoServer "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/helm/argo-workflow.yaml"
    click ArgoTemplates "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/argo-workflow-template.tf"
    click ArgoTemplates "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/argo-workflow/workflowTemplate-cloud-storage-LargeNumberObjects.yaml"
    click VMAgent "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/victoriametrics.tf"
    click VMAgent "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/victoriametrics/VMAgent.yaml"
    click VMPodScrape "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/victoriametrics/VMPodScrape.yaml"
    click VMSingle "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/victoriametrics/VMSingle.yaml"
    click Grafana "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/helm/grafana.yaml"
    click ConfigMaps "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/configmaps.tf"
    click Secrets "https://github.com/chandras/data-sync/blob/main/high-capacity-data-migration-tool-main/secrets.tf"

    %% Styles
    classDef infra fill:#FFA500,stroke:#333,stroke-width:1px
    classDef control fill:#1E90FF,stroke:#333,stroke-width:1px
    classDef data fill:#32CD32,stroke:#333,stroke-width:1px
    classDef monitor fill:#20B2AA,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
