How nalluraj/job-market-pipeline works
This platform turns North American job postings into salary and skill analytics. The daily workflow fetches postings, stores them in MySQL, extracts skills, copies normalized records to Snowflake, transforms them with dbt, and serves analytics in a Streamlit dashboard. Airflow dispatches the stages; the dashboard reads the five documented marts. The README establishes the broad warehouse and dbt flow, while the excerpts show ingestion, loading, scheduling, and dashboard queries; dbt model-to-model details are not sampled.
Main entry points: staging Staging Models · dashboard.py Streamlit Dashboard · job_fetcher.py Job Fetcher · job_pipeline_dag.py Airflow Scheduler
Components 17
Ingestion
- Job Fetcher
ingestion/job_fetcher.py - Raw JSON Cache
ingestion/job_fetcher.py - MySQL Job Store
storage/schema.sql - Skill Extractor
ingestion/skill_extractor.py - Snowflake Loader
ingestion/snowflake_loader.py
Warehouse Analytics
- Snowflake Warehouse
- Staging Models
transformation/job_market_dbt/models/staging - Salary by Role
transformation/job_market_dbt/models/marts/mart_salary_by_role.sql - Top Skills
transformation/job_market_dbt/models/marts/mart_top_skills.sql - Salary by Location
transformation/job_market_dbt/models/marts/mart_salary_by_location.sql - Salary by Company
transformation/job_market_dbt/models/marts/mart_salary_by_company.sql - Estimated Salary
transformation/job_market_dbt/models/marts/mart_estimated_salary.sql
Dashboard Serving
- Streamlit Dashboard
serving/dashboard.py
Orchestration
- Airflow Scheduler
orchestration/dags/job_pipeline_dag.py
Other components
- Dashboard User
- JSearch API
- S3 Raw Storage
Connections 24
- Airflow Scheduler → Job Fetcher: dispatches
orchestration/dags/job_pipeline_dag.py - Airflow Scheduler → Skill Extractor: dispatches
orchestration/dags/job_pipeline_dag.py - Airflow Scheduler → Snowflake Loader: dispatches
orchestration/dags/job_pipeline_dag.py - Airflow Scheduler → Staging Models: runs dbt
orchestration/dags/job_pipeline_dag.py - Job Fetcher → JSearch API: queries
ingestion/job_fetcher.py - Job Fetcher → Raw JSON Cache: writes JSON
ingestion/job_fetcher.py - Job Fetcher → S3 Raw Storage: uploads raw
ingestion/job_fetcher.py - Job Fetcher → MySQL Job Store: inserts postings
ingestion/job_fetcher.py - Skill Extractor → MySQL Job Store: reads and writes
ingestion/skill_extractor.py - Snowflake Loader → MySQL Job Store: reads records
ingestion/snowflake_loader.py - Snowflake Loader → Snowflake Warehouse: loads records
ingestion/snowflake_loader.py - Staging Models → Snowflake Warehouse: transforms warehouse data
README.md - Staging Models → Salary by Role: builds marts
README.md - Staging Models → Top Skills: builds marts
README.md - Staging Models → Salary by Location: builds marts
README.md - Staging Models → Salary by Company: builds marts
README.md - Staging Models → Estimated Salary: builds marts
README.md - Dashboard User → Streamlit Dashboard: views
README.md - Streamlit Dashboard → Salary by Role: queries
serving/dashboard.py - Streamlit Dashboard → Top Skills: queries
serving/dashboard.py - Streamlit Dashboard → Salary by Location: queries
serving/dashboard.py - Streamlit Dashboard → Salary by Company: queries
serving/dashboard.py - Streamlit Dashboard → Estimated Salary: queries
serving/dashboard.py - Streamlit Dashboard → Snowflake Warehouse: connects to
serving/dashboard.py