# pavanku7/adf-realtimescenarios architecture

> Architecture diagram of the GitHub repository pavanku7/adf-realtimescenarios, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pavanku7/adf-realtimescenarios
- Repository: https://github.com/pavanku7/adf-realtimescenarios

## Overview

##### Project type and purpose
- From the README (“ADF-RealtimeScenarios”) and ZIP names (e.g., “OnPrem_datacopy_support_live.zip”, “Incremental-copy_support_live.zip”), this is an Azure Data Factory (ADF)–based data-integration solution.
- Purpose: move data in near-real time from an on-premises SQL Server into Azure Data Lake Storage (ADLS) and/or Azure SQL Database, using incremental copy patterns and multi-folder support.

##### File structure
- Top-level files:
- README.md – configuration and connectivity steps.
- Several ZIP packages, each containing pipeline definitions or ARM templates for different scenarios:
– OnPrem_datacopy_support_live.zip
– Incremental-copy_support_live.zip
– MultiFolder-IncrementCopy_support_live.zip
- These suggest three key pattern implementations: basic on-prem copy, incremental data loads, and multi-folder incremental loads.
- No “frontend” or “backend” code directories—this is a pure data-pipeline project.

##### README for additional insights
- Focuses on SQL Server network setup (TCP/IP, firewall rules, authentication).
- Mentions “public network access is disabled” and how to add firewall rules for ADF.
- Implies use of a Self-Hosted Integration Runtime (IR) on the on-prem network, and Azure Data Factory in the cloud.

##### Detailed capture of essential elements
- Do not omit the Integration Runtime—this is the critical secure connector.
- Firewall and port requirements (TCP 1433, UDP 1434 outbound/inbound rules).
- Azure networking controls (public network access settings, IP ranges).
- Briefly note configuration steps (mixed-mode auth, enabling TCP/IP) as callouts.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Corporate Network"
        MSSQL["MSSQLSERVER"]:::corp
        FW["Firewall<br>(TCP 1433, UDP 1434)"]:::corp
        IR["Self-Hosted IR"]:::corp
        README["README.md"]:::doc
    end

    subgraph "Azure Cloud"
        ADF["Azure Data Factory"]:::azure
        subgraph "Pipelines"
            OP["OnPremCopy"]:::pipeline
            IC["IncrementalCopy"]:::pipeline
            MF["MultiFolderIncrementalCopy"]:::pipeline
        end
        ADLS["ADLS Gen2"]:::storage
        SQLDB["Azure SQL DB"]:::storage
    end

    MSSQL -->|"TCP 1433"| FW
    FW --> IR
    IR -->|"HTTPS 443"| ADF
    ADF -->|"Copy to ADLS"| ADLS
    ADF -->|"Load to SQL DB"| SQLDB

    click OP "https://github.com/pavanku7/adf-realtimescenarios/blob/main/OnPrem_datacopy_support_live.zip"
    click IC "https://github.com/pavanku7/adf-realtimescenarios/blob/main/Incremental-copy_support_live.zip"
    click MF "https://github.com/pavanku7/adf-realtimescenarios/blob/main/MultiFolder-IncrementCopy_support_live.zip"
    click README "https://github.com/pavanku7/adf-realtimescenarios/blob/main/README.md"

    classDef corp fill:#D6EAF8,stroke:#2980B9,color:#154360
    classDef azure fill:#F2F3F4,stroke:#7D3C98,color:#4A235A
    classDef pipeline fill:#FAD7A0,stroke:#D35400,color:#7E5109
    classDef storage fill:#D5F5E3,stroke:#1D8348,color:#145A32
    classDef doc fill:#FCF3CF,stroke:#B7950B,color:#665A3C
```
