How oussama-elmaachy/job_market_pipeline-databricks-pyspark works
Job Market Pipeline collects job postings from a REST API, lands raw responses in Bronze, cleans and enriches them in Silver, and produces Gold analytics datasets and dimensions for reporting. The README documents the medallion workflow and example analyses; sampled transformations show the Bronze merge, Silver processing and dimension merges, and Gold aggregation. Notebook internals and SQL query implementations were not sampled, so those links rely only on documented capabilities; no finer notebook-to-module call wiring is asserted.
Main entry points: silver_transformations.py Silver transforms · gold_transformations.py Gold aggregation · api_response.py API client · ingestion_transformations.py Bronze ingestion
Components 16
API ingestion
- API client
src/utils/api_response.py - Bronze ingestion
src/transformations/ingestion_transformations.py - Pipeline notebooks
notebooks
Medallion processing
- Bronze Delta table
src/defintion_tables/def_bronze_table.sql - Silver transforms
src/transformations/silver_transformations.py - Spark session
src/utils/spark_utils.py - Pipeline config
src/utils/config.py - Silver Delta table
src/defintion_tables/def_silver_table.sql
Analytics modeling
- Gold aggregation
src/transformations/gold_transformations.py - Gold Delta table
src/defintion_tables/def_gold_table.sql - Job dimensions
src/defintion_tables - Location enrichment
src/utils/geo_location.py
Analytics consumption
- Analytics queries
src/sql - Power BI reporting
dashbord
Other components
- Pipeline operator
- Job Search API
Connections 17
- Pipeline operator → Pipeline notebooks: runs in order
README.md - API client → Job Search API: requests postings
src/utils/api_response.py - Pipeline notebooks → API client: fetches postings
- API client → Bronze ingestion: returns raw jobs
- Bronze ingestion → Bronze Delta table: merges records
src/transformations/ingestion_transformations.py - Bronze Delta table → Silver transforms: cleans postings
README.md - Silver transforms → Spark session: gets session
src/transformations/silver_transformations.py - Silver transforms → Pipeline config: reads table names
src/transformations/silver_transformations.py - Silver transforms → Silver Delta table: merges records
src/transformations/silver_transformations.py - Silver transforms → Job dimensions: merges dimensions
src/transformations/silver_transformations.py - Silver transforms → Location enrichment: enriches locations
src/transformations/silver_transformations.py - Silver Delta table → Gold aggregation: aggregates for analysis
README.md - Gold aggregation → Spark session: gets session
src/transformations/gold_transformations.py - Gold aggregation → Pipeline config: reads table names
src/transformations/gold_transformations.py - Gold aggregation → Gold Delta table: writes aggregates
src/transformations/gold_transformations.py - Gold Delta table → Analytics queries: supports analysis
README.md - Analytics queries → Power BI reporting: feeds reporting