# keiver31/etl_transacciones_fraudulente_powerbi architecture

> Architecture diagram of the GitHub repository keiver31/etl_transacciones_fraudulente_powerbi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/keiver31/etl_transacciones_fraudulente_powerbi
- Repository: https://github.com/keiver31/etl_transacciones_fraudulente_powerbi

## Overview

##### Main Components
- Data Source – Kaggle dataset (provided as a zipped file, kaggle.zip)
- AWS Lambda Functions – Two separate Lambda scripts for:
- Extracting data from Kaggle (scripts/Lambda_Kaggle_Extraccion.py)
- Saving the processed file to S3 (scripts/Lambda_Save_File_S3.py)
- AWS S3 Bucket – Storage for raw and transformed data (documentation in docs/Bucket_S3.md)
- AWS Glue – Comprising a Crawler (docs/Crawler_S3_to_Glue-DataCatalog.md) and a Job that transforms and loads data from S3 to RDS (docs/DataCatalog_to_RDS.md)
- AWS RDS MySQL – Database where the transformed data is stored (docs/RDS_MySQL.md)
- Power BI – For creating interactive dashboards and visualizations (the Fraude_Bancario.pbix file and sample images in the images folder)
- IAM – Underpins security and permissions across AWS services

##### Flow and Interactions
- Data Origination: The initial data set is acquired from Kaggle.
- Data Extraction: A Lambda function fetches and cleans the Kaggle dataset.
- Storage: The cleaned dataset is saved in an S3 bucket.
- Data Transformation: AWS Glue (via its crawler and ETL job) reads the S3 data, transforms it, and prepares it for insertion.
- Data Loading: The transformed data is loaded into an RDS MySQL instance.
- Data Consumption: Power BI connects to the RDS (using JDBC as described in the README) for interactive data analysis and visualization.

##### Architectural Patterns and Design Principles
- Modular ETL Pipeline: Each stage (extraction, storage, transformation, loading) is handled as a separate AWS service.
- Decoupled Components: AWS Lambda, S3, Glue, and RDS work independently, improving scalability and maintainability.
- Cloud-Native Integration: The design leverages the integration capabilities of AWS, such as IAM for secure service interaction.
- Clear Separation of Concerns: Data ingestion is separated from data processing and visualization, allowing the maintenance or upgrade of individual components without impacting the entire pipeline.

## Diagram (Mermaid)

```mermaid
graph TD
    dsKaggle["Kaggle Data Source (Dataset)"]:::external
    lambdaExtract["Lambda_Kaggle_Extraccion"]:::aws
    lambdaSave["Lambda_Save_File_S3"]:::aws
    s3bucket["AWS S3 Bucket"]:::aws
    subgraph "AWS Glue"
        glueCrawler["Glue Crawler (DataCatalog)"]:::aws
        glueJob["Glue Job (Data Transformation & Load)"]:::aws
    end
    rds["AWS RDS MySQL"]:::aws
    powerBI["Power BI Dashboard"]:::external
    iam["IAM Policies"]:::aws

    dsKaggle -->|"extracts"| lambdaExtract
    lambdaExtract -->|"cleans_and_triggers"| lambdaSave
    lambdaSave -->|"saves_to"| s3bucket
    s3bucket -->|"triggers"| glueCrawler
    glueCrawler -->|"catalogs"| glueJob
    glueJob -->|"loads_into"| rds
    rds -->|"connects_via_JDBC"| powerBI

    iam -.-> lambdaExtract
    iam -.-> lambdaSave
    iam -.-> s3bucket
    iam -.-> glueCrawler
    iam -.-> glueJob
    iam -.-> rds

    click dsKaggle "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/kaggle.zip"
    click lambdaExtract "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/scripts/Lambda_Kaggle_Extraccion.py"
    click lambdaSave "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/scripts/Lambda_Save_File_S3.py"
    click s3bucket "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/docs/Bucket_S3.md"
    click glueCrawler "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/docs/Crawler_S3_to_Glue-DataCatalog.md"
    click glueJob "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/docs/DataCatalog_to_RDS.md"
    click rds "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/docs/RDS_MySQL.md"
    click powerBI "https://github.com/keiver31/etl_transacciones_fraudulente_powerbi/blob/master/Fraude_Bancario.pbix"

    classDef aws fill:#D4EFDF,stroke:#117A65,stroke-width:2px;
    classDef external fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
```
