How mrnidhi/yt-analysis works
Project Type & Purpose
• This is a serverless, event-driven ETL pipeline on AWS that ingests YouTube trending CSV files, cleans/transforms them, catalogs the results, and exposes data to BI tools for analytics.
File Structure
• cloudformation/
↳ Infrastructure as Code (S3 buckets, Lambda, Glue, Athena setup)
• lambda/
↳ Python Lambda for data cleaning
• glue/
↳ Python ETL job for transformation & cataloging
• athena/
↳ Sample SQL queries for analytics
• dataset/
↳ Sample CSV data
• docs/
README Insights
• Lists AWS services, data zones (raw/clean/artifacts), and BI integration.
Main Components & Interactions
a. Data Input
• YouTube CSV files (dataset/USvideos.csv) → Uploads into S3 Raw Zone
b. AWS Storage
• S3 Raw Zone (“raw/”)
• S3 Clean Zone (“clean/”)
• S3 Artifacts (for Lambda/Glue code)
c. Processing
• Lambda Function (lambda/lambda_function.py) triggered by S3 event → cleans records → writes to Clean Zone
• Glue Crawler & Glue ETL Job (glue/glue_job_etl.py) on Clean Zone → catalog tables + optionally convert to Parquet
d. Analytics
• Athena Queries (athena/query*.sql) against Glue-populated Data Catalog
• BI Tools (QuickSight / Power BI / Tableau) connect via JDBC/ODBC to Athena
Architectural Patterns & Principles
• Serverless & event-driven (S3 → Lambda → Glue)
• Data zoning (raw vs. clean vs. cataloged)
• Infrastructure as Code with CloudFormation
• Separation of concerns: ingestion, cleaning, transformation, querying
Final Checklist
[ ] Tiles for each component with:
‐ Name & brief description