# ayanhussain81/aws-youtube-etl-pipeline architecture

> Architecture diagram of the GitHub repository ayanhussain81/aws-youtube-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayanhussain81/aws-youtube-etl-pipeline
- Repository: https://github.com/ayanhussain81/aws-youtube-etl-pipeline

## Overview

This repository describes a serverless AWS data lake for regional YouTube trending statistics. The workflow uploads CSV and category JSON inputs to S3, transforms the two data types through separate Lambda and Glue paths, catalogs the cleansed outputs, and exposes joined analysis in Athena and QuickSight. The graph preserves the documented crawler/catalog stages and the alternate region-filtered Glue job. Relationships for unsampled AWS configuration and dashboard setup are based on the README; source excerpts confirm the Lambda and Glue-job operations.

Main entry points: [`Project_Code/Project_Code/Code/video8 Amazon S3 CLI copy commands.sh`](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video8%20Amazon%20S3%20CLI%20copy%20commands.sh) (S3 upload commands), [`Project_Code/Project_Code/Code/video22 Spark code Glue job.py`](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video22%20Spark%20code%20Glue%20job.py) (Statistics Spark ETL), [`Project_Code/Project_Code/Code/video24 Spark code Glue job with pushdown predicate.py`](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video24%20Spark%20code%20Glue%20job%20with%20pushdown%20predicate.py) (Filtered Spark ETL), [`Project_Code/Project_Code/Code/video17 lambda function.py`](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video17%20lambda%20function.py) (Category JSON Lambda)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_data["Regional CSV and JSON"]
  node_upload["S3 upload commands"]
  node_raw[("Raw S3 bucket")]
end

subgraph group_reference["Reference ETL"]
  node_lambda["Category JSON Lambda"]
  node_clean[("Cleansed S3 bucket")]
end

subgraph group_statistics["Statistics ETL"]
  node_rawcrawler["Raw Glue crawler"]
  node_catalog[("Glue Data Catalog")]
  node_spark["Statistics Spark ETL"]
  node_pushdown["Filtered Spark ETL"]
end

subgraph group_analytics["Analytics"]
  node_cleancrawler["Cleansed Glue crawler"]
  node_athena["Athena analysis"]
  node_quicksight["QuickSight dashboard"]
end

node_operator(("Data operator"))

node_operator -->|"runs upload"| node_upload
node_data -->|"provides files"| node_upload
node_upload -->|"copies to"| node_raw
node_raw -->|"triggers on JSON"| node_lambda
node_lambda -->|"writes Parquet"| node_clean
node_raw -->|"crawls CSVs"| node_rawcrawler
node_rawcrawler -->|"registers raw tables"| node_catalog
node_spark -->|"reads raw table"| node_catalog
node_spark -->|"writes Parquet"| node_clean
node_pushdown -->|"reads selected regions"| node_catalog
node_pushdown -->|"writes Parquet"| node_clean
node_clean -->|"crawls cleansed data"| node_cleancrawler
node_cleancrawler -->|"registers cleansed tables"| node_catalog
node_athena -->|"joins catalog tables"| node_catalog
node_athena -->|"supplies analysis"| node_quicksight

click node_data "https://github.com/ayanhussain81/aws-youtube-etl-pipeline/tree/master/Data"
click node_upload "https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video8%20Amazon%20S3%20CLI%20copy%20commands.sh"
click node_lambda "https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video17%20lambda%20function.py"
click node_spark "https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video22%20Spark%20code%20Glue%20job.py"
click node_pushdown "https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video24%20Spark%20code%20Glue%20job%20with%20pushdown%20predicate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data,node_upload,node_raw toneBlue
class node_lambda,node_clean toneAmber
class node_rawcrawler,node_catalog,node_spark,node_pushdown toneMint
class node_cleancrawler,node_athena,node_quicksight toneRose
class node_operator toneIndigo
```

## Components

### Ingestion

- [Regional CSV and JSON](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/tree/master/Data): `Data`
- [S3 upload commands](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video8%20Amazon%20S3%20CLI%20copy%20commands.sh): `Project_Code/Project_Code/Code/video8 Amazon S3 CLI copy commands.sh`
- **Raw S3 bucket**

### Reference ETL

- [Category JSON Lambda](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video17%20lambda%20function.py): `Project_Code/Project_Code/Code/video17 lambda function.py`
- **Cleansed S3 bucket**

### Statistics ETL

- **Raw Glue crawler**
- **Glue Data Catalog**
- [Statistics Spark ETL](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video22%20Spark%20code%20Glue%20job.py): `Project_Code/Project_Code/Code/video22 Spark code Glue job.py`
- [Filtered Spark ETL](https://github.com/ayanhussain81/aws-youtube-etl-pipeline/blob/master/Project_Code/Project_Code/Code/video24%20Spark%20code%20Glue%20job%20with%20pushdown%20predicate.py): `Project_Code/Project_Code/Code/video24 Spark code Glue job with pushdown predicate.py`

### Analytics

- **Cleansed Glue crawler**
- **Athena analysis**
- **QuickSight dashboard**

### Other components

- **Data operator**

## Connections

- Data operator → S3 upload commands: runs upload (evidence: `README.md`)
- Regional CSV and JSON → S3 upload commands: provides files (evidence: `README.md`)
- S3 upload commands → Raw S3 bucket: copies to (evidence: `README.md`)
- Raw S3 bucket → Category JSON Lambda: triggers on JSON (evidence: `README.md`)
- Category JSON Lambda → Cleansed S3 bucket: writes Parquet (evidence: `Project_Code/Project_Code/Code/video17 lambda function.py`)
- Raw S3 bucket → Raw Glue crawler: crawls CSVs (evidence: `README.md`)
- Raw Glue crawler → Glue Data Catalog: registers raw tables (evidence: `README.md`)
- Statistics Spark ETL → Glue Data Catalog: reads raw table (evidence: `Project_Code/Project_Code/Code/video22 Spark code Glue job.py`)
- Statistics Spark ETL → Cleansed S3 bucket: writes Parquet (evidence: `Project_Code/Project_Code/Code/video22 Spark code Glue job.py`)
- Filtered Spark ETL → Glue Data Catalog: reads selected regions (evidence: `Project_Code/Project_Code/Code/video24 Spark code Glue job with pushdown predicate.py`)
- Filtered Spark ETL → Cleansed S3 bucket: writes Parquet (evidence: `Project_Code/Project_Code/Code/video24 Spark code Glue job with pushdown predicate.py`)
- Cleansed S3 bucket → Cleansed Glue crawler: crawls cleansed data (evidence: `README.md`)
- Cleansed Glue crawler → Glue Data Catalog: registers cleansed tables (evidence: `README.md`)
- Athena analysis → Glue Data Catalog: joins catalog tables (evidence: `README.md`)
- Athena analysis → QuickSight dashboard: supplies analysis (evidence: `README.md`)
