How sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics works
Project Type and Purpose
• This is a hybrid data engineering pipeline built with Python scripts and SQL, designed to ingest, process, and serve data using a Bronze-Silver-Gold architecture on Microsoft Fabric and Azure Cloud.
• It supports real-time ingestion from on-premise sources, multi-layered transformation, and delivery of analytics-ready datasets to Power BI.
File Structure Analysis
• Top-level directories:
• dw_schema_scripts: SQL DDL/DML scripts for creating and dropping tables in the data warehouse.
• src: Python packages for pipeline logic (pipeline_scripts), triggers (watchdog monitor), and utility functions (schema management, validation).
• tests: unit tests for code quality.
• Patterns:
• “pipeline_scripts” contains functions to build Bronze and Silver views.
• “triggers” folder holds the watchdog service for file monitoring.
• “utils” encapsulates database parameter handling, schema operations, metadata retrieval, and data validations.
• Configuration: requirements.txt lists dependencies; readme.md describes deployment steps and architecture.
README Insights
• Confirms a Bronze-Silver-Gold layered model: raw data ingestion → validation/transformation → enrichment and analytics.
• References key components: Watchdog service, Azure Data Lake (Parquet), Fabric SQL Database, Dataflow Gen2, Fabric Data Warehouse, and Power BI.
• Emphasizes real-time capabilities, hybrid on-premise/cloud integration, and scalable ETL.
Key Components
a. Data Sources
• On-premise systems (IoT devices, logs, CRM/ERP, external APIs)
b. Watchdog-Monitor Trigger
• Python service that watches local directories and triggers ingestion jobs.
c. Bronze Layer (Raw)
• Azure Data Lake storage (Parquet files)
d. Silver Layer (Validated & Cleansed)
• Microsoft Fabric SQL Database
• SQL workflows performing validation, cleansing, business logic.
e. Gold Layer (Enriched & Aggregated)
• Fabric Dataflow Gen2 and Fabric Data Warehouse
• Dataflow transformations and final warehouse tables.
f. Analytics & Reporting
• Power BI dashboards connecting to Fabric Data Warehouse
g. External Integrations
• Azure Cloud services (Data Lake, Fabric, Data Warehouse)
• Optional connectors to external APIs or ML services for future enhancements.
Relationships and Data Flow
• From Bronze Layer → Fabric SQL Database (Silver) → Dataflow Gen2 → Fabric Data Warehouse (Gold)
• From Gold → Power BI
Architectural Patterns and Principles
• Layered Bronze/Silver/Gold design for separation of concerns.
• Event-driven ingestion via a file-watcher trigger.
• Serverless/cloud-native storage and processing for scalability and cost optimization.
• Modular code structure (pipeline scripts vs. utilities vs. triggers).
Technologies and Frameworks
• Python-based orchestration (src/pipeline_scripts, src/triggers, src/utils)
• SQL scripts for schema management (dw_schema_scripts)
• Azure Data Lake (Parquet)
• Microsoft Fabric SQL Database
• Dataflow Gen2
• Fabric Data Warehouse
• Power BI for visualization