# nada912/tracks-record architecture

> Architecture diagram of the GitHub repository nada912/tracks-record, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nada912/tracks-record
- Repository: https://github.com/nada912/tracks-record

## Overview

##### Main Components
- Raw Data Storage
- Local text files (downloaded from Teams)
- Databricks “data” folder (DBFS)
- Compute / Processing Layer
- Databricks cluster running PySpark (Notebook)
- Data Warehouse
- Supabase PostgreSQL instance (public.tracks_record fact + dimension tables)
- BI / Presentation Layer
- Power BI desktop connecting via PostgreSQL connector
- Users / Clients
- Data engineers (upload files, run notebooks)
- BI Analysts (view dashboards)

##### Architectural Patterns & Design Principles
- ETL Pipeline: Clear separation of ingestion, transformation, and load steps
- Star Schema: central “tracks_record” fact table surrounded by Year, Week, Listener, Track, Album, Artist dimensions
- Modular Notebook: single PySpark notebook implements extraction, cleaning, enrichment, schema enforcement
- Scalability: Runs on a managed Databricks Spark cluster
- Maintainability: Reproducible via notebook import/run; DB schema enforces data integrity

##### Technologies & Frameworks
- Databricks (Spark cluster, DBFS, Notebook)
- PySpark (RDD <-> DataFrame API, Spark SQL functions)
- Supabase PostgreSQL (host: aws-0-eu-central-1.pooler.supabase.com:6543)
- JDBC driver (org.postgresql.Driver)
- Power BI Desktop (PostgreSQL connector, .pbix file)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Users
    subgraph "Users"
        user1((Data Engineers)):::users
        user2((BI Analysts)):::users
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        localFiles[(Local Text Files)]:::ingestion
        dbfs[(DBFS 'data' folder)]:::ingestion
    end

    %% Processing Layer
    subgraph "Processing Layer"
        cluster(("Databricks Cluster")):::processing
        notebook[/Process Track Records.ipynb/]:::processing
        notebookDoc[/Process Track Records.html/]:::processing
        readme[/README.md/]:::documentation
    end

    %% Data Warehouse Layer
    subgraph "Data Warehouse Layer"
        pg[(Supabase PostgreSQL)]:::storage
        fact[(tracks_record Fact Table)]:::storage
        dimYear[(Year Dimension)]:::storage
        dimWeek[(Week Dimension)]:::storage
        dimListener[(Listener Dimension)]:::storage
        dimTrack[(Track Dimension)]:::storage
        dimAlbum[(Album Dimension)]:::storage
        dimArtist[(Artist Dimension)]:::storage
    end

    %% BI Layer
    subgraph "BI / Presentation Layer"
        pbx[/Tracks Dashboard.pbix/]:::bi
        report[/Report Dashboard.pdf/]:::bi
    end

    %% Data Flow
    user1 -->|upload files| localFiles
    localFiles -->|Create table→upload| dbfs
    dbfs -->|RDD→DataFrame| notebook
    cluster -->|runs notebook| notebook
    notebook -->|JDBC write| pg
    pg -->|PostgreSQL connector| pbx
    pbx -->|interactive dashboards| user2

    %% Star Schema relationships
    fact -->|FK Year| dimYear
    fact -->|FK Week| dimWeek
    fact -->|FK Listener| dimListener
    fact -->|FK Track| dimTrack
    fact -->|FK Album| dimAlbum
    fact -->|FK Artist| dimArtist

    %% Click Events
    click notebook "https://github.com/nada912/tracks-record/blob/main/Process Track Records.ipynb"
    click notebookDoc "https://github.com/nada912/tracks-record/blob/main/Process Track Records.html"
    click readme "https://github.com/nada912/tracks-record/blob/main/README.md"
    click pbx "https://github.com/nada912/tracks-record/blob/main/Tracks Dashboard.pbix"
    click report "https://github.com/nada912/tracks-record/blob/main/Report Dashboard.pdf"

    %% Styles
    classDef ingestion fill:#c2f0c2,stroke:#333,stroke-width:1px
    classDef processing fill:#add8e6,stroke:#333,stroke-width:1px
    classDef storage fill:#f4a460,stroke:#333,stroke-width:1px
    classDef bi fill:#dda0dd,stroke:#333,stroke-width:1px
    classDef users fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef documentation fill:#ffffff,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
