# chipmuon/dataengineer-youtube-factor-analysis architecture

> Architecture diagram of the GitHub repository chipmuon/dataengineer-youtube-factor-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chipmuon/dataengineer-youtube-factor-analysis
- Repository: https://github.com/chipmuon/dataengineer-youtube-factor-analysis

## Overview

##### Main Components
- Data Sources: Represent the raw data provided by the Kaggle dataset (separate sources for CSV and JSON videos data).
- Ingestion / ETL Process: Indicate the use of Python-based transformation as implemented in the two Jupyter notebooks (Youtube-transformation-csv.ipynb and Youtube-transformation-json-file.ipynb). This is the “Transformation” component.
- Data Lake: Incorporate Azure Blob Storage or an equivalent centralized repository to store the ingested raw and possibly transformed data.
- Database / Data Warehouse: Add the MS SQL Server as a storage backend for structured, cleaned data that will support reporting and analytics.
- Reporting / Visualization: Include PowerBI in the design to highlight the reporting and dashboard layer that consumes the processed data.

##### Relationships and Interactions
- Flow of data from the raw sources into Azure Blob Storage.
- Scheduling/orchestration interaction of Azure Data Factory managing these pipelines.
- Final flow from the processed data into the MS SQL Server, from which PowerBI extracts data for dashboard creation.

##### Architectural Patterns and Design Principles
- Separation of concerns with clearly defined layers for ingestion, transformation/processing, storage, and reporting.
- Scalability – annotate that data processing in Azure Databricks and storage in Azure Blob and MS SQL Server are designed to scale as data volume increases.
- Security aspects (e.g., secure data management in the cloud) if deemed relevant.

##### Relevant Technologies
- Python: Used for ETL scripts/notebooks.
- Azure Components: Azure Blob Storage, Azure Databricks, Azure Data Factory.
- MS SQL Server and PowerBI for the final reporting layer.
- Existence of a “data lake” and “ETL system” which are key to the overall design.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        DS["Raw Data Sources (CSV & JSON)"]:::data_source
        BLOB["Azure Blob Storage (Data Lake)"]:::storage
    end

    %% Orchestration & ETL Layer
    subgraph "Orchestration & ETL Layer"
        ADF["Azure Data Factory"]:::orchestration
        subgraph "ETL Transformations"
            CSV["Youtube-transformation-csv.ipynb"]:::processing
            JSON["Youtube-transformation-json-file.ipynb"]:::processing
        end
        DATABRICKS["Azure Databricks"]:::processing
    end

    %% Data Warehouse & Reporting Layer
    subgraph "Data Warehouse & Reporting Layer"
        SQL["MS SQL Server"]:::database
        POWERBI["PowerBI Dashboard"]:::reporting
    end

    %% Project Documentation
    DOC["Project Overview / Documentation (README.md)"]:::documentation

    %% Data Flow Connections
    DS -->|"ingest"| BLOB
    BLOB -->|"orchestratedBy"| ADF
    ADF -->|"triggersETL"| CSV
    ADF -->|"triggersETL"| JSON
    CSV -->|"processData"| DATABRICKS
    JSON -->|"processData"| DATABRICKS
    DATABRICKS -->|"loadData"| SQL
    SQL -->|"feeds"| POWERBI

    %% Documentation connection (reference)
    DOC ---|"documents"| DS

    %% Click Events
    click CSV "https://github.com/chipmuon/dataengineer-youtube-factor-analysis/blob/main/Youtube-transformation-csv.ipynb"
    click JSON "https://github.com/chipmuon/dataengineer-youtube-factor-analysis/blob/main/Youtube-transformation-json-file.ipynb"
    click DOC "https://github.com/chipmuon/dataengineer-youtube-factor-analysis/blob/main/README.md"

    %% Styles
    classDef data_source fill:#f9e79f,stroke:#d4ac0d,stroke-width:2px;
    classDef storage fill:#aed6f1,stroke:#2874a6,stroke-width:2px;
    classDef orchestration fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef processing fill:#fadbd8,stroke:#c0392b,stroke-width:2px;
    classDef database fill:#dcdde1,stroke:#7f8fa6,stroke-width:2px;
    classDef reporting fill:#ffeaa7,stroke:#fdcb6e,stroke-width:2px;
    classDef documentation fill:#f5cba7,stroke:#e67e22,stroke-width:2px;
```
