# nucleuscloud/neosync architecture

> Architecture diagram of the GitHub repository nucleuscloud/neosync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nucleuscloud/neosync
- Repository: https://github.com/nucleuscloud/neosync

## Overview

##### Main System Components

The system architecture should be divided into these primary components:

a. Frontend Layer:
- Next.js web application (/frontend/apps/web)
- React components for UI
- API clients and SDK (/frontend/packages/sdk)

b. Backend Services:
- Management API Service (/backend)
- Worker Service (/worker)
- Authentication Service (supports multiple providers)

c. Data Storage:
- PostgreSQL database (for system state)
- Redis (for caching)
- Temporal (for workflow orchestration)

d. External Integrations:
- Database Connections (PostgreSQL, MySQL, MSSQL)
- Cloud Storage (AWS S3, GCP Cloud Storage)
- NoSQL Databases (MongoDB, DynamoDB)
- Authentication Providers

##### Component Relationships

a. Client-Server Communication:
- Frontend → Backend (Connect RPC/gRPC)
- API Authentication flow
- WebSocket connections for real-time updates

b. Backend Service Interactions:
- Management API → Worker Service
- Both services → Temporal
- Services → PostgreSQL
- Services → Redis

c. External System Connections:
- Worker → Source Databases
- Worker → Destination Databases
- Worker → Cloud Storage
- Backend → Auth Providers

##### Data Flow

Following flows:

a. Job Execution Flow:
**User Request → API → Temporal → Worker → Source DB → Transformation → Destination DB**

b. Authentication Flow:
**User → Frontend → Auth Provider → Backend → Database**

##### Key Architectural Patterns

- Event-driven architecture (Temporal workflows)
- Microservices architecture (API, Worker)
- Connection pooling for database access
- Async pipeline processing
- GitOps-based configuration

##### Infrastructure Components

Supporting systems:

- Kubernetes deployment options
- Helm charts configuration
- Monitoring and metrics (Prometheus)
- Logging (Loki)
- Tracing (OpenTelemetry)

##### Security Boundaries

Clearly mark:

- Authentication boundaries
- API authorization zones
- Data transformation security
- Connection encryption

##### Specific Implementation Details

Technical elements:

- Benthos stream processing
- JavaScript transformation engine
- Database schema management
- Job scheduling system
- Connection management

- Clear labels for all components
- Security boundary markings
- Version/environment indicators

This architecture should emphasize Neosync's core capabilities in data anonymization, synthetic data generation, and environment synchronization while showing how the system maintains data integrity and security throughout the process.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Frontend
        WebApp["Next.js Web App"]
        SDK["API Client SDK"]
    end

    subgraph BackendServices
        API["Management API Service"]
        Worker["Worker Service"]
        Auth["Authentication System"]
        ConnMgr["Connection Manager"]
        JobMgr["Job Management"]
        RBAC["Security/RBAC"]
        Metrics["Metrics/Monitoring"]
    end

    subgraph DataStorage
        PostgreSQL[(PostgreSQL)]
        Redis[(Redis Cache)]
        Temporal[("Temporal<br>Workflow Engine")]
    end

    subgraph DataSources
        direction TB
        SQL["SQL Databases"]
        Mongo["MongoDB"]
        DynamoDB["DynamoDB"]
        S3["Cloud Storage"]
    end

    subgraph TransformationLayer
        Benthos["Benthos Transformers"]
        SchemaManager["Schema Management"]
    end

    %% Frontend to Backend
    WebApp --> API
    WebApp --> SDK
    SDK --> API

    %% Backend Internal Communications
    API --> Auth
    API --> ConnMgr
    API --> JobMgr
    API --> RBAC
    API --> Temporal
    Worker --> Temporal
    Worker --> Redis
    API --> PostgreSQL
    Worker --> PostgreSQL

    %% Data Processing Flow
    Worker --> SQL
    Worker --> Mongo
    Worker --> DynamoDB
    Worker --> S3
    Worker --> Benthos
    ConnMgr --> SchemaManager

    %% External Auth
    Auth --> AuthProviders[("Auth Providers")]

    %% Styling
    classDef frontend fill:#42a5f5,stroke:#1976d2,color:white
    classDef backend fill:#66bb6a,stroke:#43a047,color:white
    classDef storage fill:#ff7043,stroke:#f4511e,color:white
    classDef transform fill:#7e57c2,stroke:#5e35b1,color:white
    classDef datasource fill:#26a69a,stroke:#00897b,color:white

    class WebApp,SDK frontend
    class API,Worker,Auth,ConnMgr,JobMgr,RBAC,Metrics backend
    class PostgreSQL,Redis,Temporal storage
    class Benthos,SchemaManager transform
    class SQL,Mongo,DynamoDB,S3 datasource

    %% Click Events
    click WebApp "https://github.com/nucleuscloud/neosync/tree/main//frontend/apps/web"
    click SDK "https://github.com/nucleuscloud/neosync/tree/main//frontend/packages/sdk"
    click API "https://github.com/nucleuscloud/neosync/tree/main//backend/cmd/mgmt"
    click Worker "https://github.com/nucleuscloud/neosync/tree/main//worker/cmd/worker"
    click Auth "https://github.com/nucleuscloud/neosync/tree/main//backend/internal/auth"
    click SQL "https://github.com/nucleuscloud/neosync/tree/main//backend/pkg/sqlconnect"
    click Mongo "https://github.com/nucleuscloud/neosync/tree/main//backend/pkg/mongoconnect"
    click DynamoDB "https://github.com/nucleuscloud/neosync/tree/main//internal/aws"
    click JobMgr "https://github.com/nucleuscloud/neosync/tree/main//backend/internal/job"
    click Benthos "https://github.com/nucleuscloud/neosync/tree/main//worker/pkg/benthos/transformers"
    click SchemaManager "https://github.com/nucleuscloud/neosync/tree/main//backend/sql/postgresql"
    click ConnMgr "https://github.com/nucleuscloud/neosync/tree/main//backend/internal/connection-manager"
    click Temporal "https://github.com/nucleuscloud/neosync/tree/main//backend/internal/temporal"
    click Redis "https://github.com/nucleuscloud/neosync/tree/main//worker/internal/redis"
    click RBAC "https://github.com/nucleuscloud/neosync/tree/main//backend/internal/ee/rbac"
    click Metrics "https://github.com/nucleuscloud/neosync/tree/main//backend/pkg/metrics"
```
