# josephmachado/cost_effective_data_pipelines architecture

> Architecture diagram of the GitHub repository josephmachado/cost_effective_data_pipelines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/josephmachado/cost_effective_data_pipelines
- Repository: https://github.com/josephmachado/cost_effective_data_pipelines

## Overview

##### Component Relationships

**[Data Sources Layer]**
- ↓
**[Processing Layer]**
- ↓
**[Storage Layer]**

##### Detailed Component Breakdown

Data Sources Layer:
- SQLite DB storing raw TPC-H data

Processing Layer:
- Create four processing nodes representing each Python script:
- exchange_data.py
- dim_parts_supplier.py
- one_big_table.py
- wide_month_supplier_metrics.py
- DuckDB engine as a central processing component

Storage Layer:
- Four output directories:
- dim_parts_supplier/
- exchange_data/
- wide_lineitem/
- wide_month_supplier_metrics/

##### Additional Elements

A technical stack sidebar showing:
- Python 3.8+
- DuckDB
- SQLite3
- Make utility

##### Performance Characteristics

A notes section indicating:
- Processing capabilities (8 core, 32GB RAM reference)
- Data volume handling (1GB-100GB)
- Performance metrics

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef source fill:#a8d1f0,stroke:#000
    classDef processor fill:#90EE90,stroke:#000
    classDef storage fill:#FFE5B4,stroke:#000
    classDef database fill:#D3D3D3,stroke:#000
    classDef api fill:#FFB6C1,stroke:#000
    classDef engine fill:#DDA0DD,stroke:#000

    %% Data Sources Layer
    subgraph Sources["Data Sources Layer"]
        TPCH["TPC-H Generator"]:::source
        COIN["Coincap API"]:::api
        SQLITE["SQLite DB (tpch.db)"]:::database
    end

    %% Processing Layer
    subgraph Processing["Processing Layer"]
        DUCK["DuckDB Engine"]:::engine
        
        subgraph Processors["Data Processors"]
            EX["Exchange Data Processor"]:::processor
            DPS["Dimension Parts Supplier Processor"]:::processor
            OBT["One Big Table Processor"]:::processor
            WMS["Wide Month Supplier Metrics Processor"]:::processor
        end
    end

    %% Storage Layer
    subgraph Storage["Storage Layer"]
        EXS["Exchange Data Storage"]:::storage
        DPSS["Dimension Parts Supplier Storage"]:::storage
        WLS["Wide Lineitem Storage"]:::storage
        WMSS["Wide Month Supplier Metrics Storage"]:::storage
    end

    %% Relationships
    TPCH --> SQLITE
    COIN --> DUCK
    SQLITE --> DUCK
    
    DUCK --> EX
    DUCK --> DPS
    DUCK --> OBT
    DUCK --> WMS
    
    EX --> EXS
    DPS --> DPSS
    OBT --> WLS
    WMS --> WMSS

    %% Click Events
    click TPCH "https://github.com/josephmachado/cost_effective_data_pipelines/tree/main/tpch-dbgen/"
    click SQLITE "https://github.com/josephmachado/cost_effective_data_pipelines/blob/main/upstream_db/tpch_DDL_DML.sql"
    click EX "https://github.com/josephmachado/cost_effective_data_pipelines/blob/main/src/data_processor/exchange_data.py"
    click DPS "https://github.com/josephmachado/cost_effective_data_pipelines/blob/main/src/data_processor/dim_parts_supplier.py"
    click OBT "https://github.com/josephmachado/cost_effective_data_pipelines/blob/main/src/data_processor/one_big_table.py"
    click WMS "https://github.com/josephmachado/cost_effective_data_pipelines/blob/main/src/data_processor/wide_month_supplier_metrics.py"
    click EXS "https://github.com/josephmachado/cost_effective_data_pipelines/tree/main/processed_data/exchange_data/"
    click DPSS "https://github.com/josephmachado/cost_effective_data_pipelines/tree/main/processed_data/dim_parts_supplier/"
    click WLS "https://github.com/josephmachado/cost_effective_data_pipelines/tree/main/processed_data/wide_lineitem/"
    click WMSS "https://github.com/josephmachado/cost_effective_data_pipelines/tree/main/processed_data/wide_month_supplier_metrics/"

    %% Legend
    subgraph Legend
        L1["Data Source"]:::source
        L2["API"]:::api
        L3["Database"]:::database
        L4["Processing Engine"]:::engine
        L5["Processor"]:::processor
        L6["Storage"]:::storage
    end
```
