How keiver31/etl_transacciones_fraudulente_powerbi works
Main Components
• Data Source – Kaggle dataset (provided as a zipped file, kaggle.zip)
• AWS Lambda Functions – Two separate Lambda scripts for:
• Extracting data from Kaggle (scripts/Lambda_Kaggle_Extraccion.py)
• Saving the processed file to S3 (scripts/Lambda_Save_File_S3.py)
• AWS S3 Bucket – Storage for raw and transformed data (documentation in docs/Bucket_S3.md)
• AWS Glue – Comprising a Crawler (docs/Crawler_S3_to_Glue-DataCatalog.md) and a Job that transforms and loads data from S3 to RDS (docs/DataCatalog_to_RDS.md)
• AWS RDS MySQL – Database where the transformed data is stored (docs/RDS_MySQL.md)
• Power BI – For creating interactive dashboards and visualizations (the Fraude_Bancario.pbix file and sample images in the images folder)
• IAM – Underpins security and permissions across AWS services
Flow and Interactions
• Data Origination: The initial data set is acquired from Kaggle.
• Data Extraction: A Lambda function fetches and cleans the Kaggle dataset.
• Storage: The cleaned dataset is saved in an S3 bucket.
• Data Transformation: AWS Glue (via its crawler and ETL job) reads the S3 data, transforms it, and prepares it for insertion.
• Data Loading: The transformed data is loaded into an RDS MySQL instance.
• Data Consumption: Power BI connects to the RDS (using JDBC as described in the README) for interactive data analysis and visualization.
Architectural Patterns and Design Principles
• Modular ETL Pipeline: Each stage (extraction, storage, transformation, loading) is handled as a separate AWS service.
• Decoupled Components: AWS Lambda, S3, Glue, and RDS work independently, improving scalability and maintainability.
• Cloud-Native Integration: The design leverages the integration capabilities of AWS, such as IAM for secure service interaction.
• Clear Separation of Concerns: Data ingestion is separated from data processing and visualization, allowing the maintenance or upgrade of individual components without impacting the entire pipeline.