# sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics architecture

> Architecture diagram of the GitHub repository sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics
- Repository: https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics

## Overview

##### Project Type and Purpose
- This is a hybrid data engineering pipeline built with Python scripts and SQL, designed to ingest, process, and serve data using a Bronze-Silver-Gold architecture on Microsoft Fabric and Azure Cloud.
- It supports real-time ingestion from on-premise sources, multi-layered transformation, and delivery of analytics-ready datasets to Power BI.

##### File Structure Analysis
- Top-level directories:
- dw_schema_scripts: SQL DDL/DML scripts for creating and dropping tables in the data warehouse.
- src: Python packages for pipeline logic (pipeline_scripts), triggers (watchdog monitor), and utility functions (schema management, validation).
- tests: unit tests for code quality.
- Patterns:
- “pipeline_scripts” contains functions to build Bronze and Silver views.
- “triggers” folder holds the watchdog service for file monitoring.
- “utils” encapsulates database parameter handling, schema operations, metadata retrieval, and data validations.
- Configuration: requirements.txt lists dependencies; readme.md describes deployment steps and architecture.

##### README Insights
- Confirms a Bronze-Silver-Gold layered model: raw data ingestion → validation/transformation → enrichment and analytics.
- References key components: Watchdog service, Azure Data Lake (Parquet), Fabric SQL Database, Dataflow Gen2, Fabric Data Warehouse, and Power BI.
- Emphasizes real-time capabilities, hybrid on-premise/cloud integration, and scalable ETL.

##### Key Components
a. Data Sources
- On-premise systems (IoT devices, logs, CRM/ERP, external APIs)
b. Watchdog-Monitor Trigger
- Python service that watches local directories and triggers ingestion jobs.
c. Bronze Layer (Raw)
- Azure Data Lake storage (Parquet files)
d. Silver Layer (Validated & Cleansed)
- Microsoft Fabric SQL Database
- SQL workflows performing validation, cleansing, business logic.
e. Gold Layer (Enriched & Aggregated)
- Fabric Dataflow Gen2 and Fabric Data Warehouse
- Dataflow transformations and final warehouse tables.
f. Analytics & Reporting
- Power BI dashboards connecting to Fabric Data Warehouse
g. External Integrations
- Azure Cloud services (Data Lake, Fabric, Data Warehouse)
- Optional connectors to external APIs or ML services for future enhancements.

##### Relationships and Data Flow
- From Bronze Layer → Fabric SQL Database (Silver) → Dataflow Gen2 → Fabric Data Warehouse (Gold)
- From Gold → Power BI

##### Architectural Patterns and Principles
- Layered Bronze/Silver/Gold design for separation of concerns.
- Event-driven ingestion via a file-watcher trigger.
- Serverless/cloud-native storage and processing for scalability and cost optimization.
- Modular code structure (pipeline scripts vs. utilities vs. triggers).

##### Technologies and Frameworks
- Python-based orchestration (src/pipeline_scripts, src/triggers, src/utils)
- SQL scripts for schema management (dw_schema_scripts)
- Azure Data Lake (Parquet)
- Microsoft Fabric SQL Database
- Dataflow Gen2
- Fabric Data Warehouse
- Power BI for visualization

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "On-Premise Systems"
        DataSources["On-Premise Data Sources"]:::ingestion
    end

    subgraph "Orchestration & Utilities"
        Watchdog["Watchdog-Monitor Trigger"]:::ingestion
        DBParam["DB Param Handling Utility"]:::processing
        SchemaOp["Schema Operations Utility"]:::processing
        Metadata["Metadata Retrieval Utility"]:::processing
        ValidateUtil["Data Validation Functions"]:::processing
        ValidationModels["Validation Models Definitions"]:::processing
    end

    DataSources --> Watchdog

    subgraph "Bronze Layer\nRaw Parquet Storage"
        BronzeScript["Bronze Ingestion Logic"]:::processing
        BronzeStorage["Azure Data Lake (Parquet)"]:::storage
    end

    Watchdog --> BronzeScript
    BronzeScript --> BronzeStorage

    subgraph "Silver Layer\nValidation & Cleansing"
        SilverScript["Silver Validation & Cleansing"]:::processing
        SilverDB["Microsoft Fabric SQL Database"]:::storage
    end

    BronzeStorage --> SilverScript
    SilverScript --> SilverDB

    subgraph "Gold Layer\nEnriched & Aggregated"
        Dataflow["Fabric Dataflow Gen2"]:::processing
        Warehouse["Fabric Data Warehouse"]:::storage
    end

    SilverDB --> Dataflow
    Dataflow --> Warehouse

    Warehouse --> PowerBI["Power BI Dashboards"]:::analytics

    subgraph "Schema Management Scripts"
        CreateMeta["Create Meta Data Table"]:::processing
        DateDim["Date Dimension Script"]:::processing
        Drop["Drop Scripts"]:::processing
    end

    CreateMeta --> Warehouse
    DateDim --> Warehouse
    Drop --> Warehouse

    Readme["Deployment & Architecture Documentation"]:::processing
    Requirements["Python Dependency Specification"]:::processing

    click Watchdog "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/triggers/watchdog-monitor_view.py"
    click BronzeScript "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/pipeline_scripts/bronze_store_view.py"
    click SilverScript "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/pipeline_scripts/silver_store_view.py"
    click CreateMeta "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/dw_schema_scripts/create_meta_data_table.sql"
    click DateDim "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/dw_schema_scripts/date_table_query.sql"
    click Drop "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/dw_schema_scripts/drop_script.sql"
    click DBParam "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/utils/db_param_view.py"
    click SchemaOp "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/utils/manage_schema_view.py"
    click Metadata "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/utils/metadata_view.py"
    click ValidateUtil "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/utils/validate_view.py"
    click ValidationModels "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/src/utils/validation_models.py"
    click Readme "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/readme.md"
    click Requirements "https://github.com/sanketrs/implementation-of-modern-data-engineering-architecture-with-fabric_analytics/blob/master/requirements.txt"

    classDef ingestion fill:#D6EAF8,stroke:#3498DB,stroke-width:2px
    classDef storage fill:#D5F5E3,stroke:#27AE60,stroke-width:2px
    classDef processing fill:#F9E79F,stroke:#D68910,stroke-width:2px
    classDef analytics fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px
```
