How chipmuon/dataengineer-youtube-factor-analysis works
Main Components
• Data Sources: Represent the raw data provided by the Kaggle dataset (separate sources for CSV and JSON videos data).
• Ingestion / ETL Process: Indicate the use of Python-based transformation as implemented in the two Jupyter notebooks (Youtube-transformation-csv.ipynb and Youtube-transformation-json-file.ipynb). This is the “Transformation” component.
• Data Lake: Incorporate Azure Blob Storage or an equivalent centralized repository to store the ingested raw and possibly transformed data.
• Database / Data Warehouse: Add the MS SQL Server as a storage backend for structured, cleaned data that will support reporting and analytics.
• Reporting / Visualization: Include PowerBI in the design to highlight the reporting and dashboard layer that consumes the processed data.
Relationships and Interactions
• Flow of data from the raw sources into Azure Blob Storage.
• Scheduling/orchestration interaction of Azure Data Factory managing these pipelines.
• Final flow from the processed data into the MS SQL Server, from which PowerBI extracts data for dashboard creation.
Architectural Patterns and Design Principles
• Separation of concerns with clearly defined layers for ingestion, transformation/processing, storage, and reporting.
• Scalability – annotate that data processing in Azure Databricks and storage in Azure Blob and MS SQL Server are designed to scale as data volume increases.
• Security aspects (e.g., secure data management in the cloud) if deemed relevant.
Relevant Technologies
• Python: Used for ETL scripts/notebooks.
• Azure Components: Azure Blob Storage, Azure Databricks, Azure Data Factory.
• MS SQL Server and PowerBI for the final reporting layer.
• Existence of a “data lake” and “ETL system” which are key to the overall design.