# vishukk1503/etl-pipeline-aws architecture

> Architecture diagram of the GitHub repository vishukk1503/etl-pipeline-aws, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vishukk1503/etl-pipeline-aws
- Repository: https://github.com/vishukk1503/etl-pipeline-aws

## Overview

##### Main Components
- ETL Orchestrator (main.py): This is the central script that invokes each ETL phase in a sequential order.
- Extract Module (src/extract.py): Responsible for connecting to Amazon Redshift, executing SQL queries, and returning raw data in a pandas DataFrame.
- Transform Module (src/transform.py): Responsible for transforming the extracted data by applying business rules, removing duplicates, and ensuring data consistency.
- Load Module (src/load_data_to_s3.py): Handles establishing a connection to Amazon S3 and writing the transformed data as a CSV file.
- Environment Configuration (.env file derived from .env.example): Supplies necessary credentials and configuration settings for authenticating connections to Redshift and S3.

##### Architectural Patterns and Design Principles
- Modular design by showcasing each ETL phase as a distinct module, helping with maintainability and potential testing.
- Sequential flow of data: Extract → Transform → Load.
- Error handling and logging within these components since they are mentioned in the README.

##### Relevant Technologies, Frameworks, or Libraries
- Amazon Redshift for the data warehouse.
- Amazon S3 for cloud storage.
- Python and its libraries (e.g., pandas) for data processing.
- Environment configuration (via .env files) to manage sensitive credentials.
- SQL for data extraction and transformation tasks.

- Step 1: Start with the ETL Orchestrator (main.py) at the center.
- Step 5: Optionally add a “Dependencies & Libraries” note to the side to indicate Python packages (as listed in requirements.txt).

## Diagram (Mermaid)

```mermaid
graph TB
    %% Nodes
    ETL["ETL Orchestrator (main.py)"]:::orchestrator
    EX["Extract Module (src/extract.py)"]:::module
    TR["Transform Module (src/transform.py)"]:::module
    LD["Load Module (src/load_data_to_s3.py)"]:::module
    ENV["Environment Variables (.env)"]:::env
    DEPS["Dependencies & Libraries (requirements.txt)"]:::note
    RS["Amazon Redshift [(Data Warehouse)]"]:::external
    S3["Amazon S3 (Cloud Storage)"]:::external

    %% Subgraphs for vertical layering
    subgraph "ETL Pipeline Process"
        direction TB
        ETL
        EX
        TR
        LD
    end

    %% Connections from ETL Orchestrator to modules (sequential invocation)
    ETL -->|"invokes"| EX
    ETL -->|"invokes"| TR
    ETL -->|"invokes"| LD

    %% Sequential data flow between modules
    EX -->|"outputs"| TR
    TR -->|"processes"| LD

    %% External interactions
    EX -->|"extracts from"| RS
    LD -->|"loads to"| S3

    %% Environment dependencies (dashed arrows)
    ENV -.->|"configures"| EX
    ENV -.->|"configures"| LD

    %% Dependencies note connection
    DEPS -.->|"supports"| ETL

    %% Click Events
    click ETL "https://github.com/vishukk1503/etl-pipeline-aws/blob/main/main.py"
    click EX "https://github.com/vishukk1503/etl-pipeline-aws/blob/main/src/extract.py"
    click TR "https://github.com/vishukk1503/etl-pipeline-aws/blob/main/src/transform.py"
    click LD "https://github.com/vishukk1503/etl-pipeline-aws/blob/main/src/load_data_to_s3.py"
    click ENV "https://github.com/vishukk1503/etl-pipeline-aws/blob/main/.env.example"
    click DEPS "https://github.com/vishukk1503/etl-pipeline-aws/blob/main/requirements.txt"

    %% Styles
    classDef orchestrator fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef module fill:#B4E391,stroke:#333,stroke-width:2px;
    classDef external fill:#F7A278,stroke:#333,stroke-width:2px;
    classDef env fill:#D1C4E9,stroke:#333,stroke-width:2px;
    classDef note fill:#D7FFB4,stroke:#333,stroke-width:2px;
```
