# luisumina/soundwave-analytics architecture

> Architecture diagram of the GitHub repository luisumina/soundwave-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/luisumina/soundwave-analytics
- Repository: https://github.com/luisumina/soundwave-analytics

## Overview

##### Main Components
- Data Sources: Clearly indicate the CSV files (users, songs, play logs) and the Spotify API data.
- Ingestion Layer: Represent the multiple Python scripts in the “scripts/ingestion” folder (e.g., extract_users.py, extract_songs.py, extract_play_logs.py, extract_spotify.py). These are responsible for extracting raw data.
- Data Lake: Show the storage location (local file system and/or cloud storage like S3 or Azure Blob) for raw data ingestion.
- Orchestration Layer: Indicate components like Apache Airflow (with the “dags” directory as listed in the README) that manage recurring workflows.
- Streaming Components: Include the real-time event streaming with Kafka or AWS Kinesis.
- Infrastructure & Deployment: Highlight the use of containerization (Docker), infrastructure as code (Terraform), and cloud platform deployment (AWS, Azure).
- Visualization: Add the dashboarding tools (Power BI, Tableau, or Metabase) that represent the final consumption layer.

##### Diagramming the Relationships and Data Flows
- From the Data Warehouse, indicate the route to visualization/dashboards.
- Also, demonstrate the parallel or integrated streaming pipeline that pushes real-time events into a separate processing or visualization path.
- Be sure to label any cases where the data flows into multiple destinations (for instance, both Data Lake for batch storage and real-time components for streaming).

##### Architectural Patterns and Design Principles
- Layered architecture — separating raw data ingestion, transformation, structured storage, and presentation.
- Use of a dimensional modeling approach (Star Schema) to optimize analytical queries.
- If applicable, indicate the orchestration pattern (using Apache Airflow’s DAGs for scheduling and dependency management).
- Hints of scalability through the adoption of cloud storage and streaming services.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    subgraph "Data Sources"
        DS1["CSV Files (users, songs, play logs)"]:::datasource
        DS2["Spotify API Data"]:::datasource
    end

    %% Ingestion & Streaming Layer
    subgraph "Ingestion & Streaming"
        IN1["Data Ingestion Scripts"]:::ingestion
        RS["Real-Time Streaming (Kafka/Kinesis)"]:::streaming
    end

    %% Data Lake
    DL["Data Lake (Raw Data Storage)"]:::storage

    %% Orchestration Layer
    AW["Apache Airflow (Orchestration)"]:::processing

    %% Data Warehouse (Star Schema)
    subgraph "Data Warehouse (Star Schema)"
        FT["Fact Table (fact_plays)"]:::warehouse
        D1["Dimension: Users"]:::warehouse
        D2["Dimension: Songs"]:::warehouse
        D3["Dimension: Artists"]:::warehouse
        D4["Dimension: Albums"]:::warehouse
        D5["Dimension: Devices"]:::warehouse
        D6["Dimension: Dates"]:::warehouse
        DOC["Star Schema Documentation"]:::doc
        SQL["Star Schema SQL"]:::doc
    end

    %% Visualization Layer
    VIS["Dashboards (PowerBI, Tableau, Metabase)"]:::visualization

    %% Deployment & Infrastructure
    DEP["Deployment & Infrastructure"]:::deployment

    %% Data Flow Connections
    DS1 --> IN1
    DS2 --> IN1
    DS1 --> RS
    DS2 --> RS
    IN1 --> DL
    DL --> AW
    AW --> FT
    D1 --> FT
    D2 --> FT
    D3 --> FT
    D4 --> FT
    D5 --> FT
    D6 --> FT
    FT --> VIS
    RS --> VIS

    %% Deployment support connections (dashed)
    DEP -.-> IN1
    DEP -.-> DL
    DEP -.-> AW
    DEP -.-> RS
    DEP -.-> FT

    %% Click Events
    click IN1 "https://github.com/luisumina/soundwave-analytics/tree/main/scripts/ingestion"
    click DOC "https://github.com/luisumina/soundwave-analytics/blob/main/docs/modeling/DWH_star_schema_v1.md"
    click SQL "https://github.com/luisumina/soundwave-analytics/blob/main/docs/modeling/DWH_star_schema_v1.sql"
    click DEP "https://github.com/luisumina/soundwave-analytics/blob/main/docs/architecture.md"

    %% Styles
    classDef datasource fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef ingestion fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef streaming fill:#AFEEEE,stroke:#000,stroke-width:2px;
    classDef storage fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef processing fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef warehouse fill:#F4A460,stroke:#000,stroke-width:2px;
    classDef doc fill:#EEE8AA,stroke:#000,stroke-width:2px;
    classDef visualization fill:#FF7F50,stroke:#000,stroke-width:2px;
    classDef deployment fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
