pashazanoosi/uk-job-market-intelligence-platform

Purpose and entry points

This repository is a UK job-market intelligence pipeline: it acquires Adzuna job listings, normalizes and stores them in PostgreSQL, enriches descriptions with LLM-derived skills, produces analytics/snapshots, and exposes results through Streamlit and Power BI.

Primary execution surfaces:

• Full pipeline orchestration: run_pipeline.py

• Collection-oriented scheduled run: run_collection_pipeline.py, .github/workflows/collection_pipeline.yml

• Streamlit application: app/app.py

• Power BI artifact: powerbi/uk_job_market_intelligence_dashboard.pbix

Collection and raw-job boundary

The ingestion boundary is the external Adzuna Job Search API. Collection code retrieves listings, while raw-to-structured transformation separates API-shaped records from the internal job model.

• src/data/get_jobs.py

• src/data/clean_jobs.py

• src/data/transform_jobs.py

The resulting master job dataset can also be persisted as a parquet artifact, providing a file-level handoff/recovery point:

• src/data/master_jobs.py

• src/data/save_master.py

• src/utils/uk_jobs_master.parquet

Database persistence and schema

PostgreSQL, hosted through Neon per the README, is the system of record for structured job and skill intelligence. SQLAlchemy/database connection code mediates application access; schema and migration paths define the durable data contract.

• src/database/connection.py

• src/database/schema.sql

• src/data/load_to_database.py

Historical or legacy movement is handled separately:

• src/database/migrate_data.py

AI enrichment stage

LLM operations use the Groq API to turn unstructured descriptions into skills and usable market narrative. This stage depends on stored/cleaned job text and should be treated as an external-service, non-deterministic enrichment boundary.

• src/ai/extract_skills.py

• src/ai/classify_skills.py

• src/ai/suggest_taxonomy.py

Generated market reporting is a related output path:

• src/ai/generate_market_report.py

Analytics and historical snapshots

Analytics modules aggregate database/enriched data into demand, salary, market, and skills-oriented datasets. Snapshot creation establishes the historical-trend layer rather than relying only on current listings.

• src/analytics/market_analysis.py

• src/analytics/skill_demand_analysis.py

• src/analytics/create_snapshots.py

Database-side views and snapshot tables are part of the analytics-serving contract:

• sql/analytics_views.sql

• sql/create_snapshot_tables.sql

Presentation layer

Streamlit is the interactive operational analytics surface, organized into page-level views for overview, skills, market, trends, and pipeline health. It likely reads prepared analytics datasets rather than participating in ingestion.

• app/app.py

• app/pages/1_Overview.py

• app/pages/5_Pipeline.py

Power BI is the separate executive BI consumer of the analytics database/model:

• powerbi/uk_job_market_intelligence_dashboard.pbix

Operations and notifications

GitHub Actions schedules or automates collection. Environment configuration supplies API/database/LLM credentials, while utility integrations support operational alerts and optional file exchange.

• .github/workflows/collection_pipeline.yml

• .env.example

• src/utils/telegram_alert.py

• src/utils/google_drive.py