# diegopereira93/sparkfy architecture

> Architecture diagram of the GitHub repository diegopereira93/sparkfy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/diegopereira93/sparkfy
- Repository: https://github.com/diegopereira93/sparkfy

## Overview

##### Project Overview & Key Components
- This project is an ETL pipeline designed for a data warehouse specifically supporting Sparkify’s music streaming analytics. Data in JSON format stored on S3 is processed using Python scripts, transformed according to a star schema, and loaded into a Redshift database.
- Cloud Storage (S3): Hosting raw JSON log files and song metadata.
- ETL Pipeline: Including modules such as etl.py and utility scripts (utils.py, sql_queries.py) that extract, transform, and load data.
- Database and Schema: Redshift cluster containing staging tables (staging_events, staging_songs) and dimension/fact tables (songplays, users, songs, artists, time) following a star schema.
- Cluster Management: Scripts like manage_cluster.py and create_tables.py that set up, configure, and manage the Redshift cluster.
- Analytics Layer: The component running run_analytics.py, which executes predefined analytical queries.

##### Analyzing the File Structure
- Top-level files such as create_tables.py, etl.py, run_analytics.py, and manage_cluster.py indicate separate responsibilities:
– create_tables.py for creating/updating the database schema.
– etl.py for data extraction, transformation, and loading.
– run_analytics.py for executing analytics queries.
– manage_cluster.py for handling cluster operations (setting up and tearing down the Redshift cluster).
- Configuration files (dwh.cfg.example, .env.example) and requirements.txt specify dependencies and AWS/S3 settings.
- Utility code is encapsulated within utils.py and sql_queries.py, contributing to the ETL and analytical process.

a. And Label Main Components:
- Staging and Analytical Data Warehouse:
- Inside, indicate “Staging Tables” (staging_events, staging_songs) feeding into the star schema.
- Cluster Management: Represent the management tools (manage_cluster.py and create_tables.py) as a component that controls the Redshift cluster’s setup and teardown.

b. Relationships & Data Flow:

##### Architectural Patterns & Design Principles
- The project leverages a star schema design—a proven pattern for data warehousing that simplifies complex queries and speeds up aggregations.
- ETL segmentation: The logical separation of data extraction, transformation, and loading allows for clearer error handling and performance monitoring.
- Modularity: The separation of concerns is evident with utility scripts, separate query dictionaries, and distinct modules for cluster management versus analytics querying.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Configuration"
        CONF_dwh["dwh.cfg.example"]:::configuration
        CONF_env[".env.example"]:::configuration
    end

    subgraph "S3 Data Sources"
        S3_log["S3 (JSON Log Files)"]:::dataSource
        S3_song["S3 (Song Metadata Files)"]:::dataSource
    end

    subgraph "ETL Process"
        ETL_etl["etl.py"]:::processing
        ETL_utils["utils.py"]:::processing
        ETL_sql["sql_queries.py"]:::processing
    end

    subgraph "Redshift Database"
        RS_stage["Staging Tables"]:::database
        RS_fact["Fact Table: songplays"]:::database
        RS_users["Dimension Table: users"]:::database
        RS_songs["Dimension Table: songs"]:::database
        RS_artists["Dimension Table: artists"]:::database
        RS_time["Dimension Table: time"]:::database
    end

    subgraph "Cluster Management"
        CM_manage["manage_cluster.py"]:::management
        CM_create["create_tables.py"]:::management
    end

    Analytics["run_analytics.py"]:::analytics

    %% Data Flow
    CONF_dwh --> ETL_etl
    CONF_env --> ETL_etl

    S3_log --> ETL_etl
    S3_song --> ETL_etl

    ETL_etl --> RS_stage

    RS_stage --> RS_fact
    RS_fact --> RS_users
    RS_fact --> RS_songs
    RS_fact --> RS_artists
    RS_fact --> RS_time

    CM_manage --> RS_stage
    CM_create --> RS_stage

    Analytics --> RS_fact

    %% Click Events
    click ETL_etl "https://github.com/diegopereira93/sparkfy/blob/main/etl.py"
    click ETL_utils "https://github.com/diegopereira93/sparkfy/blob/main/utils.py"
    click ETL_sql "https://github.com/diegopereira93/sparkfy/blob/main/sql_queries.py"
    click CM_manage "https://github.com/diegopereira93/sparkfy/blob/main/manage_cluster.py"
    click CM_create "https://github.com/diegopereira93/sparkfy/blob/main/create_tables.py"
    click Analytics "https://github.com/diegopereira93/sparkfy/blob/main/run_analytics.py"
    click CONF_dwh "https://github.com/diegopereira93/sparkfy/blob/main/dwh.cfg.example"
    click CONF_env "https://github.com/diegopereira93/sparkfy/blob/main/.env.example"

    %% Styles
    classDef dataSource fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef processing fill:#FDFD96,stroke:#000,stroke-width:2px;
    classDef database fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef management fill:#FFB347,stroke:#000,stroke-width:2px;
    classDef analytics fill:#FF6961,stroke:#000,stroke-width:2px;
    classDef configuration fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
