How upt-faing-epis/proyecto-si885-2025-i-u2-grupo_fv_fr_fm works
Project Type and Purpose
• This is a data-analytics platform combining an ETL/data-pipeline on AWS with a Power BI dashboard.
• It ingests job-offer data, stores and cleans it in DynamoDB, runs verification and export steps, and presents an interactive dashboard (.pbix).
File-Structure Analysis
• Top-level:
• .github/workflows – CI/CD for tests and releases
• infra – Terraform IAC for AWS resources
• DataWare – Python ETL/streaming scripts and configs
• Programa – standalone data-cleaning app + requirements.txt
• DashBoard Proyecto.pbix – Power BI report
• Documentation folders (FD01–FD05, presentacion.md)
• Patterns:
• “infra” delineates infrastructure provisioning
• “DataWare” holds ingestion, storage, cleaning, export
• Python scripts orchestrate AWS Kinesis, DynamoDB, Lambda, and Power BI API calls
README Insights
• Purpose: Monitor and analyze Peru’s tech job-market trends.
• Core deliverable: interactive Power BI dashboard
Main Components & Relationships
a. Data Sources
• CSV files (ofertas_trabajo.csv, “okey” folder)
• (Optionally external APIs or file drops)
b. Ingestion Layer (Streaming)
• WriteKinesisOfertas.py pushes raw records into AWS Kinesis Data Stream
• Kinesis-DynamoDB.yaml defines the Kinesis → DynamoDB streaming delivery
c. Storage & Cleaning
• DynamoDB table holds raw job offers
• AWS Lambda (lambda.py) or limpiar_dynamodb.py cleans or filters stale data
d. Verification & Aggregation
• verificar_datawarehouse.py validates data integrity, possibly writes to a secondary store
e. Export & Refresh
• export_to_powerbi.py calls Power BI REST APIs to push datasets
• auto_refresh_powerbi.py triggers dataset refreshes on a schedule
f. Visualization
• Power BI Dashboard (DashBoard Proyecto.pbix) consumed by end users
g. Infrastructure as Code
• infra/main.tf provisions Kinesis streams, DynamoDB tables, Lambda functions, IAM roles
Architectural Patterns & Design Principles
• Serverless ETL pipeline (Kinesis + Lambda + DynamoDB)
• Infrastructure as code (Terraform)
• Separation of concerns: ingestion, storage, cleaning, export, visualization
• Event-driven data flow with scheduled jobs for refresh
Detailed Steps for Creation
2. Next, place “AWS Kinesis Data Stream” inside an “AWS” cloud. Connect source → Kinesis.
7. Finally, display “Power BI Dashboard” accessed by end-users.