# nalluraj/job-market-pipeline architecture

> Architecture diagram of the GitHub repository nalluraj/job-market-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nalluraj/job-market-pipeline
- Repository: https://github.com/nalluraj/job-market-pipeline
- Video tour (about a minute): https://gitdiagram.com/nalluraj/job-market-pipeline/video

## Overview

This platform turns North American job postings into salary and skill analytics. The daily workflow fetches postings, stores them in MySQL, extracts skills, copies normalized records to Snowflake, transforms them with dbt, and serves analytics in a Streamlit dashboard. Airflow dispatches the stages; the dashboard reads the five documented marts. The README establishes the broad warehouse and dbt flow, while the excerpts show ingestion, loading, scheduling, and dashboard queries; dbt model-to-model details are not sampled.

Main entry points: [`transformation/job_market_dbt/models/staging`](https://github.com/nalluraj/job-market-pipeline/tree/main/transformation/job_market_dbt/models/staging) (Staging Models), [`serving/dashboard.py`](https://github.com/nalluraj/job-market-pipeline/blob/main/serving/dashboard.py) (Streamlit Dashboard), [`ingestion/job_fetcher.py`](https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/job_fetcher.py) (Job Fetcher), [`orchestration/dags/job_pipeline_dag.py`](https://github.com/nalluraj/job-market-pipeline/blob/main/orchestration/dags/job_pipeline_dag.py) (Airflow Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_fetch["Job Fetcher<br/>[job_fetcher.py]"]
  node_cache["Raw JSON Cache<br/>[job_fetcher.py]"]
  node_mysql[("MySQL Job Store<br/>[schema.sql]")]
  node_skills["Skill Extractor<br/>[skill_extractor.py]"]
  node_loader["Snowflake Loader"]
end

subgraph group_warehouse["Warehouse Analytics"]
  node_snowflake[("Snowflake Warehouse")]
  node_dbt_staging["Staging Models"]
  node_salary_role["Salary by Role"]
  node_top_skills["Top Skills"]
  node_salary_location["Salary by Location"]
  node_salary_company["Salary by Company"]
  node_estimated_salary["Estimated Salary"]
end

subgraph group_serving["Dashboard Serving"]
  node_dashboard["Streamlit Dashboard<br/>[dashboard.py]"]
end

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow Scheduler"]
end

node_analyst(("Dashboard User"))
node_jsearch{{"JSearch API"}}
node_s3[("S3 Raw Storage")]

node_airflow -->|"dispatches"| node_fetch
node_airflow -->|"dispatches"| node_skills
node_airflow -->|"dispatches"| node_loader
node_airflow -->|"runs dbt"| node_dbt_staging
node_fetch -->|"queries"| node_jsearch
node_fetch -->|"writes JSON"| node_cache
node_fetch -->|"uploads raw"| node_s3
node_fetch -->|"inserts postings"| node_mysql
node_skills -->|"reads and writes"| node_mysql
node_loader -->|"reads records"| node_mysql
node_loader -->|"loads records"| node_snowflake
node_dbt_staging -->|"transforms warehouse data"| node_snowflake
node_dbt_staging -->|"builds marts"| node_salary_role
node_dbt_staging -->|"builds marts"| node_top_skills
node_dbt_staging -->|"builds marts"| node_salary_location
node_dbt_staging -->|"builds marts"| node_salary_company
node_dbt_staging -->|"builds marts"| node_estimated_salary
node_analyst -->|"views"| node_dashboard
node_dashboard -->|"queries"| node_salary_role
node_dashboard -->|"queries"| node_top_skills
node_dashboard -->|"queries"| node_salary_location
node_dashboard -->|"queries"| node_salary_company
node_dashboard -->|"queries"| node_estimated_salary
node_dashboard -->|"connects to"| node_snowflake

click node_airflow "https://github.com/nalluraj/job-market-pipeline/blob/main/orchestration/dags/job_pipeline_dag.py"
click node_fetch "https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/job_fetcher.py"
click node_cache "https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/job_fetcher.py"
click node_mysql "https://github.com/nalluraj/job-market-pipeline/blob/main/storage/schema.sql"
click node_skills "https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/skill_extractor.py"
click node_loader "https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/snowflake_loader.py"
click node_dbt_staging "https://github.com/nalluraj/job-market-pipeline/tree/main/transformation/job_market_dbt/models/staging"
click node_salary_role "https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_salary_by_role.sql"
click node_top_skills "https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_top_skills.sql"
click node_salary_location "https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_salary_by_location.sql"
click node_salary_company "https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_salary_by_company.sql"
click node_estimated_salary "https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_estimated_salary.sql"
click node_dashboard "https://github.com/nalluraj/job-market-pipeline/blob/main/serving/dashboard.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fetch,node_cache,node_mysql,node_skills,node_loader,node_analyst toneBlue
class node_snowflake,node_dbt_staging,node_salary_role,node_top_skills,node_salary_location,node_salary_company,node_estimated_salary,node_s3 toneAmber
class node_dashboard,node_jsearch toneMint
class node_airflow toneRose
```

## Components

### Ingestion

- [Job Fetcher](https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/job_fetcher.py): `ingestion/job_fetcher.py`
- [Raw JSON Cache](https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/job_fetcher.py): `ingestion/job_fetcher.py`
- [MySQL Job Store](https://github.com/nalluraj/job-market-pipeline/blob/main/storage/schema.sql): `storage/schema.sql`
- [Skill Extractor](https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/skill_extractor.py): `ingestion/skill_extractor.py`
- [Snowflake Loader](https://github.com/nalluraj/job-market-pipeline/blob/main/ingestion/snowflake_loader.py): `ingestion/snowflake_loader.py`

### Warehouse Analytics

- **Snowflake Warehouse**
- [Staging Models](https://github.com/nalluraj/job-market-pipeline/tree/main/transformation/job_market_dbt/models/staging): `transformation/job_market_dbt/models/staging`
- [Salary by Role](https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_salary_by_role.sql): `transformation/job_market_dbt/models/marts/mart_salary_by_role.sql`
- [Top Skills](https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_top_skills.sql): `transformation/job_market_dbt/models/marts/mart_top_skills.sql`
- [Salary by Location](https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_salary_by_location.sql): `transformation/job_market_dbt/models/marts/mart_salary_by_location.sql`
- [Salary by Company](https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_salary_by_company.sql): `transformation/job_market_dbt/models/marts/mart_salary_by_company.sql`
- [Estimated Salary](https://github.com/nalluraj/job-market-pipeline/blob/main/transformation/job_market_dbt/models/marts/mart_estimated_salary.sql): `transformation/job_market_dbt/models/marts/mart_estimated_salary.sql`

### Dashboard Serving

- [Streamlit Dashboard](https://github.com/nalluraj/job-market-pipeline/blob/main/serving/dashboard.py): `serving/dashboard.py`

### Orchestration

- [Airflow Scheduler](https://github.com/nalluraj/job-market-pipeline/blob/main/orchestration/dags/job_pipeline_dag.py): `orchestration/dags/job_pipeline_dag.py`

### Other components

- **Dashboard User**
- **JSearch API**
- **S3 Raw Storage**

## Connections

- Airflow Scheduler → Job Fetcher: dispatches (evidence: `orchestration/dags/job_pipeline_dag.py`)
- Airflow Scheduler → Skill Extractor: dispatches (evidence: `orchestration/dags/job_pipeline_dag.py`)
- Airflow Scheduler → Snowflake Loader: dispatches (evidence: `orchestration/dags/job_pipeline_dag.py`)
- Airflow Scheduler → Staging Models: runs dbt (evidence: `orchestration/dags/job_pipeline_dag.py`)
- Job Fetcher → JSearch API: queries (evidence: `ingestion/job_fetcher.py`)
- Job Fetcher → Raw JSON Cache: writes JSON (evidence: `ingestion/job_fetcher.py`)
- Job Fetcher → S3 Raw Storage: uploads raw (evidence: `ingestion/job_fetcher.py`)
- Job Fetcher → MySQL Job Store: inserts postings (evidence: `ingestion/job_fetcher.py`)
- Skill Extractor → MySQL Job Store: reads and writes (evidence: `ingestion/skill_extractor.py`)
- Snowflake Loader → MySQL Job Store: reads records (evidence: `ingestion/snowflake_loader.py`)
- Snowflake Loader → Snowflake Warehouse: loads records (evidence: `ingestion/snowflake_loader.py`)
- Staging Models → Snowflake Warehouse: transforms warehouse data (evidence: `README.md`)
- Staging Models → Salary by Role: builds marts (evidence: `README.md`)
- Staging Models → Top Skills: builds marts (evidence: `README.md`)
- Staging Models → Salary by Location: builds marts (evidence: `README.md`)
- Staging Models → Salary by Company: builds marts (evidence: `README.md`)
- Staging Models → Estimated Salary: builds marts (evidence: `README.md`)
- Dashboard User → Streamlit Dashboard: views (evidence: `README.md`)
- Streamlit Dashboard → Salary by Role: queries (evidence: `serving/dashboard.py`)
- Streamlit Dashboard → Top Skills: queries (evidence: `serving/dashboard.py`)
- Streamlit Dashboard → Salary by Location: queries (evidence: `serving/dashboard.py`)
- Streamlit Dashboard → Salary by Company: queries (evidence: `serving/dashboard.py`)
- Streamlit Dashboard → Estimated Salary: queries (evidence: `serving/dashboard.py`)
- Streamlit Dashboard → Snowflake Warehouse: connects to (evidence: `serving/dashboard.py`)