# san089/udacity-data-engineering-projects architecture

> Architecture diagram of the GitHub repository san089/udacity-data-engineering-projects, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/san089/udacity-data-engineering-projects
- Repository: https://github.com/san089/udacity-data-engineering-projects

## Overview

- Data Modeling with Cassandra: Illustrate the ETL pipeline that processes CSV event data and models it into Apache Cassandra. Use the notebook and CSV files as indicators of the data flow.
- Data Warehouse on AWS (Redshift): Create components showing AWS S3 (data landing zone) and a Redshift cluster. Include scripts like Redshift_Cluster_IaC.py along with configuration files (logging.ini, CloudFormation markdown files) which manage infrastructure-as-code.
- Data Lake: Show an AWS-based data lake architecture, where raw data (log-data JSON files and song_data JSONs under Data_Lake) is ingested from S3. Also include the Spark jobs (ETL operations) as a processing layer.
- API Data to Postgres: Show a smaller pipeline component that makes API calls to fetch data from Yelp (via request.py, businesssearch.py) and loads that data into Postgres.
- Supporting AWS Services: Include the AWS_Services directory and CloudFormation templates which describe how various AWS resources are set up (for example, for Airflow Livy setup or Redshift).

##### Relationships and Data Flow
- Start from Data Sources (JSON log files, CSV files, API endpoints) and progress through ETL pipelines.
- How data moves from raw files in Data Lake or source APIs into transformation layers (e.g., Python ETL scripts) and finally into target databases (Postgres, Cassandra, or Redshift).
- In the Airflow project, illustrate how Airflow DAGs trigger ETL jobs and interact with custom operators that stage data in Redshift or perform quality checks.
- Dependencies such as external libraries (e.g., Spark for processing in the Data Lake project, Apache Airflow for orchestration) and built-in scripts for creating tables and running queries.

##### Architectural Patterns and Design Principles
- Separation of concerns (each project is modular – separate directories for Postgres, Cassandra, etc.).
- For full-stack ETL applications, underline the typical extract-transform-load flow.
- If using microservices thinking, separate out API interactions (the Data_Api_to_Postgres module) as a loosely coupled component.
- Use elements such as Infrastructure-as-Code (IaC) in the Redshift_IaC and Airflow-related CloudFormation scripts to show automation in setting up environments.

##### Incorporate Relevant Technologies and Frameworks
- Clearly label components with technologies:
- PostgreSQL, Apache Cassandra, Amazon Redshift for databases.
- Apache Airflow for workflow orchestration.
- AWS for cloud services, Spark for distributed computing in the Data Lake.
- Python for ETL scripts, Jupyter Notebooks for exploratory data modeling.
- Mention integration points: the use of Python operators under the Airflow project, SQL query modules used in multiple pipelines, and configurations for AWS services.

##### Final Recommendations for the Principal Software Engineer

This detailed explanation should allow you to map out the intricate architecture of this multi-project repository accurately, clearly delineating the individual systems while also showing how they integrate into the larger data engineering framework. Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Modeling with Postgres"
        PM_ETL["ETL Pipeline (etl.py, main.py)"]:::etl
        PM_SQL["SQL Modules (sql_queries.py, create_tables.py)"]:::etl
        PM_DB["Postgres Database"]:::storage
        PM_ETL --> PM_SQL
        PM_SQL --> PM_DB
    end

    subgraph "Data Modeling with Apache Cassandra"
        CA_ETL["ETL Pipeline (Notebook, CSV event data)"]:::etl
        CA_DB["Cassandra Database"]:::storage
        CA_ETL --> CA_DB
    end

    subgraph "Data Warehouse on AWS (Redshift)"
        DW_S3["AWS S3 Landing Zone"]:::storage
        DW_ETL["ETL Pipeline (Redshift scripts)"]:::etl
        DW_DB["Redshift Cluster"]:::storage
        DW_IaC["IaC Scripts (Redshift_Cluster_IaC.py)"]:::etl
        DW_S3 --> DW_ETL
        DW_ETL --> DW_DB
        DW_IaC --> DW_DB
    end

    subgraph "Data Lake"
        DL_S3["Raw Data (log-data, song_data)"]:::storage
        DL_ETL["Spark ETL Job (etl.py)"]:::etl
        DL_S3 --> DL_ETL
    end

    subgraph "Data Pipelines with Airflow"
        AF_DAG["Airflow DAGs"]:::orchestration
        AF_PLUGINS["Custom Plugins & Operators"]:::orchestration
        AF_CFG["Configuration Files"]:::orchestration
        AF_DAG --> AF_PLUGINS
        AF_PLUGINS --> AF_CFG
    end

    subgraph "API Data to Postgres"
        API_CALL["API Module (request.py, businesssearch.py)"]:::etl
        API_DB["Postgres Database"]:::storage
        API_CALL --> API_DB
    end

    subgraph "Supporting AWS Services"
        AWS_INFRA["AWS Infrastructure (CloudFormation templates)"]:::storage
    end

    %% Inter-project data flow triggered by Airflow
    AF_DAG -->|"Triggers_ETL"| PM_ETL
    AF_DAG -->|"Triggers_ETL"| CA_ETL
    AF_DAG -->|"Triggers_ETL"| DW_ETL
    AF_DAG -->|"Triggers_ETL"| DL_ETL
    AF_DAG -->|"Triggers_API"| API_CALL

    %% Click Events
    click PM_ETL "https://github.com/san089/udacity-data-engineering-projects/tree/master/Data_Modeling_with_Postgres"
    click CA_ETL "https://github.com/san089/udacity-data-engineering-projects/tree/master/Data_Modeling_with_Apache_Cassandra"
    click DW_ETL "https://github.com/san089/udacity-data-engineering-projects/tree/master/Data_Warehouse"
    click DW_IaC "https://github.com/san089/udacity-data-engineering-projects/blob/master/Redshift_Cluster_IaC.py"
    click DL_ETL "https://github.com/san089/udacity-data-engineering-projects/tree/master/Data_Lake"
    click AF_DAG "https://github.com/san089/udacity-data-engineering-projects/tree/master/Airflow_Data_Pipelines"
    click API_CALL "https://github.com/san089/udacity-data-engineering-projects/tree/master/Data_Api_to_Postgres"
    click AWS_INFRA "https://github.com/san089/udacity-data-engineering-projects/tree/master/AWS_Services"

    %% Styles
    classDef storage fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef etl fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef orchestration fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
