How nada912/tracks-record works
Main Components
• Raw Data Storage
• Local text files (downloaded from Teams)
• Databricks “data” folder (DBFS)
• Compute / Processing Layer
• Databricks cluster running PySpark (Notebook)
• Data Warehouse
• Supabase PostgreSQL instance (public.tracks_record fact + dimension tables)
• BI / Presentation Layer
• Power BI desktop connecting via PostgreSQL connector
• Users / Clients
• Data engineers (upload files, run notebooks)
• BI Analysts (view dashboards)
Architectural Patterns & Design Principles
• ETL Pipeline: Clear separation of ingestion, transformation, and load steps
• Star Schema: central “tracks_record” fact table surrounded by Year, Week, Listener, Track, Album, Artist dimensions
• Modular Notebook: single PySpark notebook implements extraction, cleaning, enrichment, schema enforcement
• Scalability: Runs on a managed Databricks Spark cluster
• Maintainability: Reproducible via notebook import/run; DB schema enforces data integrity
Technologies & Frameworks
• Databricks (Spark cluster, DBFS, Notebook)
• PySpark (RDD <-> DataFrame API, Spark SQL functions)
• Supabase PostgreSQL (host: aws-0-eu-central-1.pooler.supabase.com:6543)
• JDBC driver (org.postgresql.Driver)
• Power BI Desktop (PostgreSQL connector, .pbix file)