# mrnidhi/yt-analysis architecture

> Architecture diagram of the GitHub repository mrnidhi/yt-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mrnidhi/yt-analysis
- Repository: https://github.com/mrnidhi/yt-analysis

## Overview

##### Project Type & Purpose
- This is a serverless, event-driven ETL pipeline on AWS that ingests YouTube trending CSV files, cleans/transforms them, catalogs the results, and exposes data to BI tools for analytics.

##### File Structure
- cloudformation/
↳ Infrastructure as Code (S3 buckets, Lambda, Glue, Athena setup)
- lambda/
↳ Python Lambda for data cleaning
- glue/
↳ Python ETL job for transformation & cataloging
- athena/
↳ Sample SQL queries for analytics
- dataset/
↳ Sample CSV data
- docs/

##### README Insights
- Lists AWS services, data zones (raw/clean/artifacts), and BI integration.

##### Main Components & Interactions
a. Data Input
- YouTube CSV files (dataset/USvideos.csv) → Uploads into S3 Raw Zone
b. AWS Storage
- S3 Raw Zone (“raw/”)
- S3 Clean Zone (“clean/”)
- S3 Artifacts (for Lambda/Glue code)
c. Processing
- Lambda Function (lambda/lambda_function.py) triggered by S3 event → cleans records → writes to Clean Zone
- Glue Crawler & Glue ETL Job (glue/glue_job_etl.py) on Clean Zone → catalog tables + optionally convert to Parquet
d. Analytics
- Athena Queries (athena/query*.sql) against Glue-populated Data Catalog
- BI Tools (QuickSight / Power BI / Tableau) connect via JDBC/ODBC to Athena

##### Architectural Patterns & Principles
- Serverless & event-driven (S3 → Lambda → Glue)
- Data zoning (raw vs. clean vs. cataloged)
- Infrastructure as Code with CloudFormation
- Separation of concerns: ingestion, cleaning, transformation, querying

##### Final Checklist
[ ] Tiles for each component with:
‐ Name & brief description

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Input"
        USCSV["<img src='dataset/csv.png' width='20'/> YouTube CSV Files"]:::data
    end

    subgraph "AWS Storage"
        S3Raw["<img src='docs/assets/s3.png' width='20'/> S3 Raw Zone (raw/)"]:::storage
        S3Clean["<img src='docs/assets/s3.png' width='20'/> S3 Clean Zone (clean/)"]:::storage
        S3Artifacts["<img src='docs/assets/s3.png' width='20'/> S3 Artifacts"]:::storage
    end

    subgraph "Processing"
        CF["<img src='docs/assets/cloudformation.png' width='20'/> CloudFormation (IaC)"]:::infra
        Lambda["<img src='docs/assets/lambda.png' width='20'/> Lambda Function (Python 3.12)"]:::compute
        GlueCrawler["<img src='docs/assets/glue.png' width='20'/> Glue Crawler"]:::compute
        GlueJob["<img src='docs/assets/glue.png' width='20'/> Glue ETL Job"]:::compute
    end

    subgraph "Analytics"
        subgraph "Athena Queries"
            Q1["query1.sql"]:::analytics
            Q2["query2.sql"]:::analytics
            Q3["query3.sql"]:::analytics
            Q4["query4.sql"]:::analytics
            Q5["query5.sql"]:::analytics
        end
        BI["<img src='docs/assets/bi.png' width='20'/> BI Tools (QuickSight/Power BI/Tableau)"]:::external
    end

    subgraph "Documentation"
        DocsArch["Architecture Diagrams"]:::docs
        DocsScreens["Screenshots"]:::docs
    end

    %% Connections
    USCSV -->|"upload"| S3Raw
    CF -->|"provisions"| S3Raw
    CF -->|"provisions"| S3Clean
    CF -->|"provisions"| S3Artifacts
    S3Raw -->|"S3 Event"| Lambda
    Lambda -->|"writes clean data"| S3Clean
    S3Clean -->|"crawl & catalog"| GlueCrawler
    GlueCrawler -->|"triggers"| GlueJob
    GlueJob -->|"populates Data Catalog"| Q1
    GlueJob --> Q2
    GlueJob --> Q3
    GlueJob --> Q4
    GlueJob --> Q5
    Q1 -->|"JDBC/ODBC"| BI
    Q2 --> BI
    Q3 --> BI
    Q4 --> BI
    Q5 --> BI
    CF -->|"docs stored in"| DocsArch
    CF --> DocsScreens

    %% Click Events
    click USCSV "https://github.com/mrnidhi/yt-analysis/blob/main/dataset/USvideos.csv"
    click CF "https://github.com/mrnidhi/yt-analysis/blob/main/cloudformation/s3-lambda-glue-athena.yaml"
    click Lambda "https://github.com/mrnidhi/yt-analysis/blob/main/lambda/lambda_function.py"
    click GlueJob "https://github.com/mrnidhi/yt-analysis/blob/main/glue/glue_job_etl.py"
    click Q1 "https://github.com/mrnidhi/yt-analysis/blob/main/athena/query1.sql"
    click Q2 "https://github.com/mrnidhi/yt-analysis/blob/main/athena/query2.sql"
    click Q3 "https://github.com/mrnidhi/yt-analysis/blob/main/athena/query3.sql"
    click Q4 "https://github.com/mrnidhi/yt-analysis/blob/main/athena/query4.sql"
    click Q5 "https://github.com/mrnidhi/yt-analysis/blob/main/athena/query5.sql"
    click DocsArch "https://github.com/mrnidhi/yt-analysis/tree/main/docs/architecture"
    click DocsScreens "https://github.com/mrnidhi/yt-analysis/tree/main/docs/assets/screenshots"

    %% Styles
    classDef data fill:#E8F5E9,stroke:#388E3C,color:#1B5E20;
    classDef storage fill:#E3F2FD,stroke:#1976D2,color:#0D47A1;
    classDef infra fill:#FFF3E0,stroke:#FB8C00,color:#E65100;
    classDef compute fill:#F3E5F5,stroke:#8E24AA,color:#4A148C;
    classDef analytics fill:#FFFDE7,stroke:#FBC02D,color:#F57F17;
    classDef external fill:#ECEFF1,stroke:#607D8B,color:#37474F;
    classDef docs fill:#F1F8E9,stroke:#7CB342,color:#33691E;
```
