How victor-antoniassi/junior_data_analyst_test_01 works
Main Components
a) Data Sources Layer:
• External Google Drive CSV source
• Local CSV file storage (data/BASE_DADOS.csv)
b) ETL Pipeline Layer:
• Prefect Orchestration Server
• ETL Processing Script (etl.py)
• Data Transformation Components
c) Storage Layer:
• DuckDB Database (db_games_sales.db)
• Processed CSV (games_sales.csv)
d) Analysis Layer:
• Jupyter Notebook Environment
• Power BI Connection Point
┌─────────────────┐
• Data Sources │ (Top Layer)
• ┬────────┘
• ↓
┌─────────────────┐
• ETL Pipeline │ (Middle Layer)
• ┬────────┘
• ↓
┌─────────────────┐
• Storage │ (Bottom Layer)
• ┬────────┘
• ↓
┌─────────────────┐
• Analysis │ (Final Layer)
• ┘
Detailed Component Representation
a) Data Sources (Blue):
• ┌─────────────────┐
• Google Drive │
• CSV File │
• ┘
b) ETL Pipeline (Green):
• ┌─────────────────────────────┐
• Prefect Server │
• ┌───────────────────────┐ │
• ETL Process │ │
• (Python/etl.py) │ │
• ┘ │
• ┘
c) Storage (Orange):
• ┌─────────────────┐ ┌─────────────────┐
• DuckDB │ │ Processed │
• Database │ │ CSV │
• ┘ └─────────────────┘
d) Analysis (Purple):
• ┌─────────────────┐ ┌─────────────────┐
• Jupyter │ │ Power BI │
• Notebook │ │ Connection │
• ┘ └─────────────────┘
Important Elements to Highlight
• Prefect's orchestration control over the ETL process
• Data transformation steps within the ETL process
• Dual output nature (DuckDB and CSV)
• Analysis tools' connections to their respective data sources