# tomassidiskis/data-engineering-project-in-aws architecture

> Architecture diagram of the GitHub repository tomassidiskis/data-engineering-project-in-aws, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tomassidiskis/data-engineering-project-in-aws
- Repository: https://github.com/tomassidiskis/data-engineering-project-in-aws

## Overview

##### Main Components
- Data Sources: Represent the raw data files (e.g., CSVs from YouTube’s dataset from Kaggle).
- Data Ingestion: Component that takes raw data and uploads it to the cloud. This can be represented by the S3 CLI commands (s3_cli_commands.sh) and points of entry for data.
- Data Lake: Amazon S3 acts as the centralized data repository where structured and semi-structured data is stored.
- ETL System:
- Pyspark Code: Denotes the transformation logic and processing of raw data. This may be executed in an AWS Glue job that leverages Spark.
- AWS Glue: Can be highlighted as the managed service to run ETL jobs.
- Serverless Compute: AWS Lambda, as indicated by lambda_function.py, to orchestrate or trigger parts of the pipeline.
- Query & Analytics: AWS Athena, which is used to run queries on the data stored in S3.
- Security & Access Management: AWS IAM (implied from the README) to manage access controls to resources.

##### Detail the Relationships and Data Flows
- Raw data is ingested via external sources (or CLI-initiated upload) into the S3 Data Lake.
- Once the data is in S3, AWS Glue (or the pyspark_code.py transformation job) processes the raw data to transform it into the required format.
- The transformed data is stored back into S3.
- AWS Lambda functions might trigger Glue jobs or execute functions in response to data changes.
- AWS Athena queries the data from S3 for analysis.
- Security components (IAM roles) govern access between these services.

##### Architectural Patterns and Design Principles
- The architecture follows a modular, serverless design which scales automatically (leveraging AWS Lambda and Glue).
- Data flow follows an ETL pattern: ingestion → transformation → storage → analysis.
- Separation of concerns: Ingestion and ETL logic are kept separate in distinct components (CLI/s3 commands, Lambda, and Pyspark code).
- The system is designed to be highly scalable (using cloud services) and secure (with IAM).

##### Relevant Technologies and Frameworks
- AWS S3 (Data Lake)
- AWS Glue (ETL Processing with PySpark)
- AWS Lambda (Orchestration/Serverless Compute)
- AWS Athena (Query Engine)
- IAM (Security/Permissions)

##### Essential Elements
- Clear labels for every component.
- Optionally, you might show “ETL Component” detailing the role of pyspark_code.py as a transformation script.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion Layer"
        A["YouTube Trending Data (Raw CSV)"]:::external
        B["S3 CLI Commands"]:::ingestion
    end

    subgraph "Data Storage Layer"
        C(["Amazon S3 - Data Lake"]):::storage
    end

    subgraph "ETL Processing Layer"
        D["AWS Glue – ETL Engine (PySpark)"]:::etl
    end

    subgraph "Serverless Compute & Orchestration"
        E["AWS Lambda – Orchestration Function"]:::compute
    end

    subgraph "Query & Analytics Layer"
        F["AWS Athena – Query Service"]:::analytics
    end

    subgraph "Security & Access Management"
        G["AWS IAM – Access Management"]:::security
    end

    %% Data Flow Connections
    A -->|"ingest"| B
    B -->|"uploadsRawData"| C
    C -->|"triggersETL"| D
    E -->|"invokesETL"| D
    D -->|"storesTransformedData"| C
    C -->|"queriedBy"| F

    %% IAM controls access to core components
    G --- B
    G --- C
    G --- D
    G --- E
    G --- F

    %% Click Events
    click B "https://github.com/tomassidiskis/data-engineering-project-in-aws/blob/main/s3_cli_commands.sh"
    click D "https://github.com/tomassidiskis/data-engineering-project-in-aws/blob/main/pyspark_code.py"
    click E "https://github.com/tomassidiskis/data-engineering-project-in-aws/blob/main/lambda_function.py"

    %% Styles
    classDef external fill:#FFCCCC,stroke:#AA0000,stroke-width:2px;
    classDef ingestion fill:#CCFFCC,stroke:#008800,stroke-width:2px;
    classDef storage fill:#CCFFFF,stroke:#006666,stroke-width:2px;
    classDef etl fill:#FFFFCC,stroke:#CCCC00,stroke-width:2px;
    classDef compute fill:#FFCC99,stroke:#FF6600,stroke-width:2px;
    classDef analytics fill:#DDA0DD,stroke:#800080,stroke-width:2px;
    classDef security fill:#E6E6FA,stroke:#6A5ACD,stroke-width:2px;
```
