How josephmachado/cost_effective_data_pipelines works
Component Relationships
[Data Sources Layer]
• ↓
[Processing Layer]
• ↓
[Storage Layer]
Detailed Component Breakdown
Data Sources Layer:
• SQLite DB storing raw TPC-H data
Processing Layer:
• Create four processing nodes representing each Python script:
• exchange_data.py
• dim_parts_supplier.py
• one_big_table.py
• wide_month_supplier_metrics.py
• DuckDB engine as a central processing component
Storage Layer:
• Four output directories:
• dim_parts_supplier/
• exchange_data/
• wide_lineitem/
• wide_month_supplier_metrics/
Additional Elements
A technical stack sidebar showing:
• Python 3.8+
• DuckDB
• SQLite3
• Make utility
Performance Characteristics
A notes section indicating:
• Processing capabilities (8 core, 32GB RAM reference)
• Data volume handling (1GB-100GB)
• Performance metrics