How pavanku7/adf-realtimescenarios works
Project type and purpose
• From the README (“ADF-RealtimeScenarios”) and ZIP names (e.g., “OnPrem_datacopy_support_live.zip”, “Incremental-copy_support_live.zip”), this is an Azure Data Factory (ADF)–based data-integration solution.
• Purpose: move data in near-real time from an on-premises SQL Server into Azure Data Lake Storage (ADLS) and/or Azure SQL Database, using incremental copy patterns and multi-folder support.
File structure
• Top-level files:
• README.md – configuration and connectivity steps.
• Several ZIP packages, each containing pipeline definitions or ARM templates for different scenarios:
– OnPrem_datacopy_support_live.zip
– Incremental-copy_support_live.zip
– MultiFolder-IncrementCopy_support_live.zip
• These suggest three key pattern implementations: basic on-prem copy, incremental data loads, and multi-folder incremental loads.
• No “frontend” or “backend” code directories—this is a pure data-pipeline project.
README for additional insights
• Focuses on SQL Server network setup (TCP/IP, firewall rules, authentication).
• Mentions “public network access is disabled” and how to add firewall rules for ADF.
• Implies use of a Self-Hosted Integration Runtime (IR) on the on-prem network, and Azure Data Factory in the cloud.
Detailed capture of essential elements
• Do not omit the Integration Runtime—this is the critical secure connector.
• Firewall and port requirements (TCP 1433, UDP 1434 outbound/inbound rules).
• Azure networking controls (public network access settings, IP ranges).
• Briefly note configuration steps (mixed-mode auth, enabling TCP/IP) as callouts.