# dskunth/etl-pipeline architecture

> Architecture diagram of the GitHub repository dskunth/etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dskunth/etl-pipeline
- Repository: https://github.com/dskunth/etl-pipeline

## Overview

##### Project Type and Purpose
- This is a data-engineering project implementing an ETL (Extract-Transform-Load) pipeline in Python.
- It connects to an Amazon Redshift data warehouse (extract), cleans and de-duplicates data (transform), then writes the cleaned CSV to an Amazon S3 bucket (load).
- It can be run locally via CLI or containerized in Docker.

##### File Structure
Top-level artifacts:
- Dockerfile/.dockerignore – container build context
- .env.example – environment variable template
- main.py – orchestrator script calling the three pipeline stages
- requirements.txt – Python dependencies
src/
- extract.py – Redshift extraction logic (SQL queries + connection)
- transform.py – in-Python dedupe & cleaning logic
- load_data_to_s3.py – S3 connection and CSV upload logic

##### README for Additional Insights
- The README confirms the three logical stages and describes both CLI and Docker-based runs.

##### Main Components
a. Orchestrator (main.py)
b. Extract Module (src/extract.py)
c. Transform Module (src/transform.py)
d. Load Module (src/load_data_to_s3.py)
e. External Systems:
- Amazon Redshift (source)
- Amazon S3 (destination)
f. Environment Config (dotenv/.env)
g. Container Boundary (Docker image/container)

##### Data Flow & Interactions
1) main.py loads environment variables →
2) calls Extract → connects to Redshift, runs SQL, retrieves raw data →
3) passes data to Transform → applies deduplication logic →
4) hands off cleaned data to Load → connects to S3, writes CSV →
5) completes pipeline run

##### Architectural Patterns & Design Principles
- Pipeline (staged data flow)
- Single Responsibility: each script handles one stage
- Config externalization: use of .env and Docker ENV

##### Technology Call-outs
- Python 3.x (requirements.txt)
- psycopg2 or Redshift driver (extract.py)
- pandas (transform.py)
- boto3 (load_data_to_s3.py)
- python-dotenv (.env handling in CLI mode)
- Docker (Dockerfile instructions)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Container" 
        direction TB
        main["Orchestrator\n(main.py)"]:::code
        subgraph "ETL Pipeline"
            direction TB
            ext["Extract Module\n(src/extract.py)"]:::code
            trans["Transform Module\n(src/transform.py)"]:::code
            load["Load Module\n(src/load_data_to_s3.py)"]:::code
        end
    end

    .env["Env Config\n(.env.example)"]:::config
    Dockerfile["Dockerfile"]:::config
    ignore[".dockerignore"]:::config
    req["requirements.txt"]:::config
    readme["README.md"]:::config

    extsys["Amazon Redshift"]:::external
    s3["Amazon S3"]:::external

    .env -->|"env vars"| main
    Dockerfile -->|"build image"| main
    req -->|"install deps"| main
    ignore -.-> main
    readme -.-> main

    main -->|"invoke"| ext
    extsys -->|"SQL result"| ext
    ext -->|"Pandas DataFrame"| trans
    trans -->|"Pandas DataFrame"| load
    load -->|"CSV"| s3

    click main "https://github.com/dskunth/etl-pipeline/blob/main/main.py"
    click ext "https://github.com/dskunth/etl-pipeline/blob/main/src/extract.py"
    click trans "https://github.com/dskunth/etl-pipeline/blob/main/src/transform.py"
    click load "https://github.com/dskunth/etl-pipeline/blob/main/src/load_data_to_s3.py"
    click .env "https://github.com/dskunth/etl-pipeline/blob/main/.env.example"
    click Dockerfile "https://github.com/dskunth/etl-pipeline/tree/main/Dockerfile"
    click ignore "https://github.com/dskunth/etl-pipeline/blob/main/.dockerignore"
    click req "https://github.com/dskunth/etl-pipeline/blob/main/requirements.txt"
    click readme "https://github.com/dskunth/etl-pipeline/blob/main/README.md"

    classDef code fill:#D6EAF8,stroke:#1B4F72,color:#1B2631;
    classDef external fill:#FDEBD0,stroke:#B9770E,color:#873600;
    classDef config fill:#E5E8E8,stroke:#566573,color:#1B2631;
    classDef container stroke-dasharray: 5 5,stroke:#566573;
```
