# vaibhavdavedev/retailsync-etl-pipeline architecture

> Architecture diagram of the GitHub repository vaibhavdavedev/retailsync-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaibhavdavedev/retailsync-etl-pipeline
- Repository: https://github.com/vaibhavdavedev/retailsync-etl-pipeline

## Overview

##### Project Type and Purpose
- RetailSync ETL Pipeline is a domain-specific data engineering solution implementing an Extract-Transform-Load (ETL) workflow.
- It extracts transaction data from Amazon Redshift, applies Python-based data quality transformations, and loads CSV output into Amazon S3 for BI and analytics.

##### File Structure
- Top-level artifacts:
- main.py – orchestrator that ties together the ETL steps
- Dockerfile & .env/.dockerignore – containerization & environment management
- requirements.txt – Python dependencies
- src/ directory with three modules:
- extract.py – Redshift connector & SQL transformations
- transform.py – Pandas-based data validation, deduplication, business logic
- load_data_to_s3.py – boto3 integration for CSV generation and S3 upload

##### README Insights
- Clearly calls out three pipeline phases: Extract (Redshift), Transform (Python), Load (S3).
[Amazon Redshift] → Extract → Transform → Load → [Amazon S3]

a. Main Components
- Amazon Redshift data warehouse
- ETL container/compute (Docker image running Python scripts)
- S3 bucket for processed output
- Configuration store (.env variables / AWS Secrets Manager)
- Orchestrator (main.py + optional scheduler/cron)

b. Relationships & Interactions
- Flow from Extract → Transform: in-memory Pandas DataFrame
- Flow from Transform → Load: temporary CSV artifact
- main.py sits above all modules, invoking them in sequence
- Environment variables feed into each module for credentials

c. Architectural Patterns & Principles
- Modular ETL pattern: clear separation of concerns (E, T, L phases)
- Containerization for reproducibility (Docker)
- Twelve-factor app: config via environment, dependencies declared, stateless compute

d. Key Technologies & Frameworks
- Python 3.x, pandas, psycopg2-binary (Redshift connectivity), boto3 (AWS SDK), python-dotenv
- Docker for packaging
- AWS services: Amazon Redshift, S3

##### Detailed Breakdown
Layer 1: Configuration
- .env / Secrets —> consumed by ETL container
Layer 2: Orchestration
- Scheduler or CLI invokes main.py
Layer 3: Extraction
- extract.py connects to Redshift → pulls tables → initial SQL transforms
Layer 4: Transformation
- transform.py cleans, deduplicates, enriches data frames
Layer 5: Loading
- load_data_to_s3.py writes CSV → uploads to S3
Layer 6: Persistence
- S3 bucket for downstream BI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        ENV[".env example"]:::config
        DOCKERFILE["Dockerfile"]:::config
        DOCKERIGNORE[".dockerignore"]:::config
        REQUIREMENTS["requirements.txt"]:::config
        README["README.md"]:::config
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        SCHED["Scheduler/CLI"]:::compute
        MAIN["main.py"]:::compute
    end

    %% Extraction Layer
    subgraph "Extraction Layer"
        REDSHIFT["Amazon Redshift"]:::source
        EXTRACT["Extract Module"]:::compute
    end

    %% Transformation Layer
    subgraph "Transformation Layer"
        TRANSFORM["Transform Module"]:::compute
    end

    %% Loading Layer
    subgraph "Loading Layer"
        LOADMODULE["Load Module"]:::compute
        CSV["CSV Artifact"]:::storage
        S3["Amazon S3"]:::storage
    end

    %% Connections
    ENV -->|consumes env vars| MAIN
    DOCKERFILE -->|container image| MAIN
    DOCKERIGNORE -->|container build| MAIN
    REQUIREMENTS -->|dependencies| MAIN
    README -->|docs| MAIN

    SCHED -->|triggers| MAIN

    MAIN -->|orchestrates| EXTRACT
    REDSHIFT -->|psycopg2/JDBC| EXTRACT
    EXTRACT -->|pandas.DataFrame| TRANSFORM
    TRANSFORM -->|CSV file| LOADMODULE
    LOADMODULE -->|writes| CSV
    LOADMODULE -->|boto3.put_object / HTTPS| S3

    %% Click events
    click MAIN "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/main.py"
    click EXTRACT "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/src/extract.py"
    click TRANSFORM "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/src/transform.py"
    click LOADMODULE "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/src/load_data_to_s3.py"
    click ENV "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/.env.example"
    click DOCKERFILE "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/tree/main/Dockerfile"
    click DOCKERIGNORE "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/.dockerignore"
    click REQUIREMENTS "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/requirements.txt"
    click README "https://github.com/vaibhavdavedev/retailsync-etl-pipeline/blob/main/README.md"

    %% Styles
    classDef source fill:#add8e6,stroke:#000
    classDef compute fill:#90ee90,stroke:#000
    classDef storage fill:#ffa500,stroke:#000
    classDef config fill:#d3d3d3,stroke:#000
```
