# oussama-elmaachy/job_market_pipeline-databricks-pyspark architecture

> Architecture diagram of the GitHub repository oussama-elmaachy/job_market_pipeline-databricks-pyspark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark
- Repository: https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark

## Overview

Job Market Pipeline collects job postings from a REST API, lands raw responses in Bronze, cleans and enriches them in Silver, and produces Gold analytics datasets and dimensions for reporting. The README documents the medallion workflow and example analyses; sampled transformations show the Bronze merge, Silver processing and dimension merges, and Gold aggregation. Notebook internals and SQL query implementations were not sampled, so those links rely only on documented capabilities; no finer notebook-to-module call wiring is asserted.

Main entry points: [`src/transformations/silver_transformations.py`](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/silver_transformations.py) (Silver transforms), [`src/transformations/gold_transformations.py`](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/gold_transformations.py) (Gold aggregation), [`src/utils/api_response.py`](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/api_response.py) (API client), [`src/transformations/ingestion_transformations.py`](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/ingestion_transformations.py) (Bronze ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["API ingestion"]
  node_api_client["API client<br/>[api_response.py]"]
  node_bronze_flow["Bronze ingestion"]
  node_notebooks["Pipeline notebooks"]
end

subgraph group_processing["Medallion processing"]
  node_bronze_store[("Bronze Delta table")]
  node_silver_flow["Silver transforms"]
  node_spark_session["Spark session<br/>[spark_utils.py]"]
  node_pipeline_config["Pipeline config<br/>[config.py]"]
  node_silver_store[("Silver Delta table")]
end

subgraph group_model["Analytics modeling"]
  node_gold_flow["Gold aggregation"]
  node_gold_store[("Gold Delta table<br/>[def_gold_table.sql]")]
  node_dimensions[("Job dimensions")]
  node_geo_enrichment["Location enrichment<br/>[geo_location.py]"]
end

subgraph group_consumption["Analytics consumption"]
  node_analytics_sql["Analytics queries"]
  node_reporting["Power BI reporting"]
end

node_operator(("Pipeline operator"))
node_api{{"Job Search API"}}

node_operator -->|"runs in order"| node_notebooks
node_api_client -->|"requests postings"| node_api
node_notebooks -.->|"fetches postings"| node_api_client
node_api_client -.->|"returns raw jobs"| node_bronze_flow
node_bronze_flow -->|"merges records"| node_bronze_store
node_bronze_store -->|"cleans postings"| node_silver_flow
node_silver_flow -->|"gets session"| node_spark_session
node_silver_flow -->|"reads table names"| node_pipeline_config
node_silver_flow -->|"merges records"| node_silver_store
node_silver_flow -->|"merges dimensions"| node_dimensions
node_silver_flow -->|"enriches locations"| node_geo_enrichment
node_silver_store -->|"aggregates for analysis"| node_gold_flow
node_gold_flow -->|"gets session"| node_spark_session
node_gold_flow -->|"reads table names"| node_pipeline_config
node_gold_flow -->|"writes aggregates"| node_gold_store
node_gold_store -->|"supports analysis"| node_analytics_sql
node_analytics_sql -.->|"feeds reporting"| node_reporting

click node_api_client "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/api_response.py"
click node_bronze_flow "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/ingestion_transformations.py"
click node_bronze_store "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/defintion_tables/def_bronze_table.sql"
click node_silver_flow "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/silver_transformations.py"
click node_spark_session "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/spark_utils.py"
click node_pipeline_config "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/config.py"
click node_silver_store "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/defintion_tables/def_silver_table.sql"
click node_gold_flow "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/gold_transformations.py"
click node_gold_store "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/defintion_tables/def_gold_table.sql"
click node_dimensions "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/src/defintion_tables"
click node_geo_enrichment "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/geo_location.py"
click node_analytics_sql "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/src/sql"
click node_reporting "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/dashbord"
click node_notebooks "https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/notebooks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_client,node_bronze_flow,node_notebooks toneBlue
class node_bronze_store,node_silver_flow,node_spark_session,node_pipeline_config,node_silver_store toneAmber
class node_gold_flow,node_gold_store,node_dimensions,node_geo_enrichment,node_api toneMint
class node_analytics_sql,node_reporting toneRose
class node_operator toneIndigo
```

## Components

### API ingestion

- [API client](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/api_response.py): `src/utils/api_response.py`
- [Bronze ingestion](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/ingestion_transformations.py): `src/transformations/ingestion_transformations.py`
- [Pipeline notebooks](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/notebooks): `notebooks`

### Medallion processing

- [Bronze Delta table](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/defintion_tables/def_bronze_table.sql): `src/defintion_tables/def_bronze_table.sql`
- [Silver transforms](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/silver_transformations.py): `src/transformations/silver_transformations.py`
- [Spark session](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/spark_utils.py): `src/utils/spark_utils.py`
- [Pipeline config](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/config.py): `src/utils/config.py`
- [Silver Delta table](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/defintion_tables/def_silver_table.sql): `src/defintion_tables/def_silver_table.sql`

### Analytics modeling

- [Gold aggregation](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/transformations/gold_transformations.py): `src/transformations/gold_transformations.py`
- [Gold Delta table](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/defintion_tables/def_gold_table.sql): `src/defintion_tables/def_gold_table.sql`
- [Job dimensions](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/src/defintion_tables): `src/defintion_tables`
- [Location enrichment](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/blob/main/src/utils/geo_location.py): `src/utils/geo_location.py`

### Analytics consumption

- [Analytics queries](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/src/sql): `src/sql`
- [Power BI reporting](https://github.com/oussama-elmaachy/job_market_pipeline-databricks-pyspark/tree/main/dashbord): `dashbord`

### Other components

- **Pipeline operator**
- **Job Search API**

## Connections

- Pipeline operator → Pipeline notebooks: runs in order (evidence: `README.md`)
- API client → Job Search API: requests postings (evidence: `src/utils/api_response.py`)
- Pipeline notebooks → API client: fetches postings
- API client → Bronze ingestion: returns raw jobs
- Bronze ingestion → Bronze Delta table: merges records (evidence: `src/transformations/ingestion_transformations.py`)
- Bronze Delta table → Silver transforms: cleans postings (evidence: `README.md`)
- Silver transforms → Spark session: gets session (evidence: `src/transformations/silver_transformations.py`)
- Silver transforms → Pipeline config: reads table names (evidence: `src/transformations/silver_transformations.py`)
- Silver transforms → Silver Delta table: merges records (evidence: `src/transformations/silver_transformations.py`)
- Silver transforms → Job dimensions: merges dimensions (evidence: `src/transformations/silver_transformations.py`)
- Silver transforms → Location enrichment: enriches locations (evidence: `src/transformations/silver_transformations.py`)
- Silver Delta table → Gold aggregation: aggregates for analysis (evidence: `README.md`)
- Gold aggregation → Spark session: gets session (evidence: `src/transformations/gold_transformations.py`)
- Gold aggregation → Pipeline config: reads table names (evidence: `src/transformations/gold_transformations.py`)
- Gold aggregation → Gold Delta table: writes aggregates (evidence: `src/transformations/gold_transformations.py`)
- Gold Delta table → Analytics queries: supports analysis (evidence: `README.md`)
- Analytics queries → Power BI reporting: feeds reporting