How vishukk1503/etl-pipeline-aws works
Main Components
• ETL Orchestrator (main.py): This is the central script that invokes each ETL phase in a sequential order.
• Extract Module (src/extract.py): Responsible for connecting to Amazon Redshift, executing SQL queries, and returning raw data in a pandas DataFrame.
• Transform Module (src/transform.py): Responsible for transforming the extracted data by applying business rules, removing duplicates, and ensuring data consistency.
• Load Module (src/load_data_to_s3.py): Handles establishing a connection to Amazon S3 and writing the transformed data as a CSV file.
• Environment Configuration (.env file derived from .env.example): Supplies necessary credentials and configuration settings for authenticating connections to Redshift and S3.
Architectural Patterns and Design Principles
• Modular design by showcasing each ETL phase as a distinct module, helping with maintainability and potential testing.
• Sequential flow of data: Extract → Transform → Load.
• Error handling and logging within these components since they are mentioned in the README.
Relevant Technologies, Frameworks, or Libraries
• Amazon Redshift for the data warehouse.
• Amazon S3 for cloud storage.
• Python and its libraries (e.g., pandas) for data processing.
• Environment configuration (via .env files) to manage sensitive credentials.
• SQL for data extraction and transformation tasks.
• Step 1: Start with the ETL Orchestrator (main.py) at the center.
• Step 5: Optionally add a “Dependencies & Libraries” note to the side to indicate Python packages (as listed in requirements.txt).