# upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm architecture

> Architecture diagram of the GitHub repository upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm
- Repository: https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm

## Overview

##### Project Type and Purpose
- This is a data-analytics platform combining an ETL/data-pipeline on AWS with a Power BI dashboard.
- It ingests job-offer data, stores and cleans it in DynamoDB, runs verification and export steps, and presents an interactive dashboard (.pbix).

##### File-Structure Analysis
- Top-level:
- .github/workflows – CI/CD for tests and releases
- infra – Terraform IAC for AWS resources
- DataWare – Python ETL/streaming scripts and configs
- Programa – standalone data-cleaning app + requirements.txt
- DashBoard Proyecto.pbix – Power BI report
- Documentation folders (FD01–FD05, presentacion.md)
- Patterns:
- “infra” delineates infrastructure provisioning
- “DataWare” holds ingestion, storage, cleaning, export
- Python scripts orchestrate AWS Kinesis, DynamoDB, Lambda, and Power BI API calls

##### README Insights
- Purpose: Monitor and analyze Peru’s tech job-market trends.
- Core deliverable: interactive Power BI dashboard

##### Main Components & Relationships
a. Data Sources
- CSV files (ofertas_trabajo.csv, “okey” folder)
- (Optionally external APIs or file drops)
b. Ingestion Layer (Streaming)
- WriteKinesisOfertas.py pushes raw records into AWS Kinesis Data Stream
- Kinesis-DynamoDB.yaml defines the Kinesis → DynamoDB streaming delivery
c. Storage & Cleaning
- DynamoDB table holds raw job offers
- AWS Lambda (lambda.py) or limpiar_dynamodb.py cleans or filters stale data
d. Verification & Aggregation
- verificar_datawarehouse.py validates data integrity, possibly writes to a secondary store
e. Export & Refresh
- export_to_powerbi.py calls Power BI REST APIs to push datasets
- auto_refresh_powerbi.py triggers dataset refreshes on a schedule
f. Visualization
- Power BI Dashboard (DashBoard Proyecto.pbix) consumed by end users
g. Infrastructure as Code
- infra/main.tf provisions Kinesis streams, DynamoDB tables, Lambda functions, IAM roles

##### Architectural Patterns & Design Principles
- Serverless ETL pipeline (Kinesis + Lambda + DynamoDB)
- Infrastructure as code (Terraform)
- Separation of concerns: ingestion, storage, cleaning, export, visualization
- Event-driven data flow with scheduled jobs for refresh

##### Detailed Steps for Creation
2. Next, place “AWS Kinesis Data Stream” inside an “AWS” cloud. Connect source → Kinesis.
7. Finally, display “Power BI Dashboard” accessed by end-users.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Raw Data Sources"
        direction TB
        CSV1["ofertas_trabajo.csv"]:::script
        CSV2["okey/ofertas_trabajo.csv"]:::script
    end

    %% AWS Cloud
    subgraph "AWS Account"
        direction TB
        Kinesis["AWS Kinesis Data Stream"]:::aws
        Dynamo["DynamoDB Table"]:::aws
        LambdaClean["Lambda: lambda.py"]:::script
    end

    %% Processing Scripts
    Verifier["verificar_datawarehouse.py"]:::script
    CleanerScript["limpiar_dynamodb.py"]:::script
    StandaloneApp["app_limpieza_ofertas.py"]:::script

    %% Export & Refresh
    Exporter["export_to_powerbi.py"]:::script
    AutoRefresh["auto_refresh_powerbi.py"]:::script

    %% Power BI
    subgraph "Power BI Service"
        direction TB
        PBIDataset["Power BI Service"]:::saas
        PBIReport["Power BI Dashboard"]:::saas
    end

    %% Infrastructure as Code
    Terraform["Terraform (infra/main.tf)"]:::infra

    %% CI/CD
    subgraph "CI/CD Pipelines"
        direction TB
        CI[".github/workflows"]:::external
    end

    %% Connections
    CSV1 -->|"Read CSV"| CSV2
    CSV2 -->|"Push record"| WriteKinesis["WriteKinesisOfertas.py"]:::script
    WriteKinesis -->|"PutRecord"| Kinesis
    Kinesis -->|"Stream to DynamoDB"| Dynamo
    Dynamo -->|"On-modify trigger"| LambdaClean
    LambdaClean -->|"Clean data"| Dynamo
    Dynamo -->|"Invoke verify"| Verifier
    Verifier -->|"Validated data"| Dynamo
    Dynamo -->|"Read for export"| Exporter
    Exporter -->|"API call: Import dataset"| PBIDataset
    AutoRefresh -->|"Trigger refresh"| PBIDataset
    PBIDataset -->|"Serve report"| PBIReport

    %% Terraform provisioning
    Terraform -.->|"provisions"| Kinesis
    Terraform -.->|"provisions"| Dynamo
    Terraform -.->|"provisions"| LambdaClean

    %% CI/CD to Terraform
    CI -->|"deploy infra"| Terraform

    %% Click Events
    click CSV1 "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/ofertas_trabajo.csv"
    click CSV2 "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/okey/ofertas_trabajo.csv"
    click WriteKinesis "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/WriteKinesisOfertas.py"
    click KinesisDeliver "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/Kinesis-DynamoDB.yaml"
    click Dynamo "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/infra/main.tf"
    click LambdaClean "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/lambda.py"
    click StandaloneApp "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/Programa/app_limpieza_ofertas.py"
    click CleanerScript "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/limpiar_dynamodb.py"
    click Verifier "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/verificar_datawarehouse.py"
    click Exporter "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/export_to_powerbi.py"
    click AutoRefresh "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DataWare/auto_refresh_powerbi.py"
    click PBIReport "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/DashBoard Proyecto.pbix"
    click Terraform "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/infra/main.tf"
    click CI "https://github.com/upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm/blob/main/.github/workflows/main.yml"

    %% Styles
    classDef aws fill:#cce5ff,stroke:#3399ff
    classDef script fill:#d4edda,stroke:#28a745
    classDef saas fill:#fff3cd,stroke:#ffecb5
    classDef infra fill:#e2e3e5,stroke:#6c757d,stroke-dasharray: 5 5
    classDef external fill:#f8d7da,stroke:#dc3545,stroke-dasharray: 2 2
```
