# pashazanoosi/uk-job-market-intelligence-platform architecture

> Architecture diagram of the GitHub repository pashazanoosi/uk-job-market-intelligence-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pashazanoosi/uk-job-market-intelligence-platform
- Repository: https://github.com/pashazanoosi/uk-job-market-intelligence-platform

## Overview

#### Purpose and entry points
This repository is a UK job-market intelligence pipeline: it acquires Adzuna job listings, normalizes and stores them in PostgreSQL, enriches descriptions with LLM-derived skills, produces analytics/snapshots, and exposes results through Streamlit and Power BI.

Primary execution surfaces:
- Full pipeline orchestration: `run_pipeline.py`
- Collection-oriented scheduled run: `run_collection_pipeline.py`, `.github/workflows/collection_pipeline.yml`
- Streamlit application: `app/app.py`
- Power BI artifact: `powerbi/uk_job_market_intelligence_dashboard.pbix`

#### Collection and raw-job boundary
The ingestion boundary is the external Adzuna Job Search API. Collection code retrieves listings, while raw-to-structured transformation separates API-shaped records from the internal job model.
- `src/data/get_jobs.py`
- `src/data/clean_jobs.py`
- `src/data/transform_jobs.py`

The resulting master job dataset can also be persisted as a parquet artifact, providing a file-level handoff/recovery point:
- `src/data/master_jobs.py`
- `src/data/save_master.py`
- `src/utils/uk_jobs_master.parquet`

#### Database persistence and schema
PostgreSQL, hosted through Neon per the README, is the system of record for structured job and skill intelligence. SQLAlchemy/database connection code mediates application access; schema and migration paths define the durable data contract.
- `src/database/connection.py`
- `src/database/schema.sql`
- `src/data/load_to_database.py`

Historical or legacy movement is handled separately:
- `src/database/migrate_data.py`

#### AI enrichment stage
LLM operations use the Groq API to turn unstructured descriptions into skills and usable market narrative. This stage depends on stored/cleaned job text and should be treated as an external-service, non-deterministic enrichment boundary.
- `src/ai/extract_skills.py`
- `src/ai/classify_skills.py`
- `src/ai/suggest_taxonomy.py`

Generated market reporting is a related output path:
- `src/ai/generate_market_report.py`

#### Analytics and historical snapshots
Analytics modules aggregate database/enriched data into demand, salary, market, and skills-oriented datasets. Snapshot creation establishes the historical-trend layer rather than relying only on current listings.
- `src/analytics/market_analysis.py`
- `src/analytics/skill_demand_analysis.py`
- `src/analytics/create_snapshots.py`

Database-side views and snapshot tables are part of the analytics-serving contract:
- `sql/analytics_views.sql`
- `sql/create_snapshot_tables.sql`

#### Presentation layer
Streamlit is the interactive operational analytics surface, organized into page-level views for overview, skills, market, trends, and pipeline health. It likely reads prepared analytics datasets rather than participating in ingestion.
- `app/app.py`
- `app/pages/1_Overview.py`
- `app/pages/5_Pipeline.py`

Power BI is the separate executive BI consumer of the analytics database/model:
- `powerbi/uk_job_market_intelligence_dashboard.pbix`

#### Operations and notifications
GitHub Actions schedules or automates collection. Environment configuration supplies API/database/LLM credentials, while utility integrations support operational alerts and optional file exchange.
- `.github/workflows/collection_pipeline.yml`
- `.env.example`
- `src/utils/telegram_alert.py`
- `src/utils/google_drive.py`

Main entry points: [`src/ai/extract_skills.py`](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/ai/extract_skills.py) (Skill extraction & classification), [`src/data/get_jobs.py`](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/get_jobs.py) (Job collector), [`run_pipeline.py`](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/run_pipeline.py) (Full pipeline runner), [`src/data/master_jobs.py`](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/master_jobs.py) (Master job dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Collection & preparation"]
  node_adzuna{{"Adzuna Job Search API<br/>external API"}}
  node_collector["Job collector<br/>ingestion module<br/>[get_jobs.py]"]
  node_normalization["Clean &amp; transform jobs<br/>transformation modules<br/>[transform_jobs.py]"]
  node_master_jobs["Master job dataset<br/>dataset builder<br/>[master_jobs.py]"]
  node_parquet["Master jobs parquet<br/>recovery artifact"]
end

subgraph group_storage["Data platform"]
  node_postgres[("Neon PostgreSQL<br/>system of record")]
  node_db_access["Database access &amp; load<br/>SQLAlchemy persistence"]
  node_schema["Database schema<br/>durable data contract<br/>[schema.sql]"]
end

subgraph group_enrichment["AI enrichment"]
  node_groq{{"Groq API<br/>LLM service"}}
  node_skills["Skill extraction &amp; classification<br/>LLM enrichment<br/>[extract_skills.py]"]
  node_market_report["Market report generator<br/>LLM reporting"]
end

subgraph group_analytics["Analytics &amp; history"]
  node_analysis["Market &amp; skill analytics<br/>analytics modules<br/>[market_analysis.py]"]
  node_snapshots["Historical snapshots<br/>trend history builder"]
  node_analytics_sql["Analytics views &amp; snapshot tables<br/>serving contract"]
end

subgraph group_presentation["Consumers"]
  node_streamlit["Streamlit analytics app<br/>interactive dashboard<br/>[app.py]"]
  node_powerbi["Power BI dashboard<br/>executive BI"]
end

subgraph group_operations["Automation &amp; operations"]
  node_pipeline["Full pipeline runner<br/>orchestrator<br/>[run_pipeline.py]"]
  node_collection_runner["Collection runner<br/>scheduled orchestrator"]
  node_github_actions["GitHub Actions schedule<br/>workflow automation"]
  node_telegram["Telegram alerts<br/>operational notification<br/>[telegram_alert.py]"]
end

node_github_actions -->|"schedules"| node_collection_runner
node_collection_runner -->|"runs"| node_collector
node_pipeline -->|"orchestrates"| node_collector
node_pipeline -->|"orchestrates"| node_skills
node_adzuna -->|"job listings"| node_collector
node_collector -->|"raw records"| node_normalization
node_normalization -->|"structured jobs"| node_master_jobs
node_master_jobs -->|"persists"| node_parquet
node_master_jobs -->|"loads"| node_db_access
node_schema -.->|"defines"| node_postgres
node_db_access -->|"writes jobs"| node_postgres
node_postgres -->|"job descriptions"| node_skills
node_skills -->|"enrichment requests"| node_groq
node_groq -->|"skills &amp; narrative"| node_skills
node_skills -->|"enriched skills"| node_postgres
node_postgres -->|"job &amp; skill data"| node_analysis
node_analysis -->|"current metrics"| node_snapshots
node_snapshots -->|"historical metrics"| node_postgres
node_analytics_sql -.->|"defines views &amp; tables"| node_postgres
node_postgres -->|"analytics datasets"| node_streamlit
node_postgres -->|"analytics model"| node_powerbi
node_postgres -->|"market data"| node_market_report
node_market_report -->|"generation request"| node_groq
node_pipeline -.->|"pipeline alerts"| node_telegram

click node_collector "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/get_jobs.py"
click node_normalization "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/transform_jobs.py"
click node_master_jobs "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/master_jobs.py"
click node_parquet "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/utils/uk_jobs_master.parquet"
click node_pipeline "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/run_pipeline.py"
click node_collection_runner "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/run_collection_pipeline.py"
click node_github_actions "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/.github/workflows/collection_pipeline.yml"
click node_db_access "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/load_to_database.py"
click node_schema "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/database/schema.sql"
click node_skills "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/ai/extract_skills.py"
click node_market_report "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/ai/generate_market_report.py"
click node_analysis "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/analytics/market_analysis.py"
click node_snapshots "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/analytics/create_snapshots.py"
click node_analytics_sql "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/sql/analytics_views.sql"
click node_streamlit "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/app/app.py"
click node_powerbi "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/powerbi/uk_job_market_intelligence_dashboard.pbix"
click node_telegram "https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/utils/telegram_alert.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_adzuna,node_collector,node_normalization,node_master_jobs,node_parquet toneBlue
class node_postgres,node_db_access,node_schema toneAmber
class node_groq,node_skills,node_market_report toneMint
class node_analysis,node_snapshots,node_analytics_sql toneRose
class node_streamlit,node_powerbi toneIndigo
class node_pipeline,node_collection_runner,node_github_actions,node_telegram toneTeal
```

## Components

### Collection & preparation

- **Adzuna Job Search API**: external API · Source boundary for UK job listings.
- [Job collector](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/get_jobs.py): ingestion module · `src/data/get_jobs.py`
- [Clean & transform jobs](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/transform_jobs.py): transformation modules · Converts API-shaped records into the internal job model. · `src/data/transform_jobs.py`
- [Master job dataset](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/master_jobs.py): dataset builder · `src/data/master_jobs.py`
- [Master jobs parquet](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/utils/uk_jobs_master.parquet): recovery artifact · File-level handoff for the master dataset. · `src/utils/uk_jobs_master.parquet`

### Data platform

- **Neon PostgreSQL**: system of record · Stores structured jobs, skills, and analytics-serving data.
- [Database access & load](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/data/load_to_database.py): SQLAlchemy persistence · `src/data/load_to_database.py`
- [Database schema](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/database/schema.sql): durable data contract · `src/database/schema.sql`

### AI enrichment

- **Groq API**: LLM service · Non-deterministic skill and narrative enrichment boundary.
- [Skill extraction & classification](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/ai/extract_skills.py): LLM enrichment · `src/ai/extract_skills.py`
- [Market report generator](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/ai/generate_market_report.py): LLM reporting · `src/ai/generate_market_report.py`

### Analytics & history

- [Market & skill analytics](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/analytics/market_analysis.py): analytics modules · Builds demand, salary, market, and skill datasets. · `src/analytics/market_analysis.py`
- [Historical snapshots](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/analytics/create_snapshots.py): trend history builder · `src/analytics/create_snapshots.py`
- [Analytics views & snapshot tables](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/sql/analytics_views.sql): serving contract · `sql/analytics_views.sql`

### Consumers

- [Streamlit analytics app](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/app/app.py): interactive dashboard · Operational views including overview, skills, trends, and pipeline health. · `app/app.py`
- [Power BI dashboard](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/powerbi/uk_job_market_intelligence_dashboard.pbix): executive BI · `powerbi/uk_job_market_intelligence_dashboard.pbix`

### Automation & operations

- [Full pipeline runner](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/run_pipeline.py): orchestrator · `run_pipeline.py`
- [Collection runner](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/run_collection_pipeline.py): scheduled orchestrator · `run_collection_pipeline.py`
- [GitHub Actions schedule](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/.github/workflows/collection_pipeline.yml): workflow automation · `.github/workflows/collection_pipeline.yml`
- [Telegram alerts](https://github.com/pashazanoosi/uk-job-market-intelligence-platform/blob/main/src/utils/telegram_alert.py): operational notification · `src/utils/telegram_alert.py`

## Connections

- GitHub Actions schedule → Collection runner: schedules
- Collection runner → Job collector: runs
- Full pipeline runner → Job collector: orchestrates
- Full pipeline runner → Skill extraction & classification: orchestrates
- Adzuna Job Search API → Job collector: job listings
- Job collector → Clean & transform jobs: raw records
- Clean & transform jobs → Master job dataset: structured jobs
- Master job dataset → Master jobs parquet: persists
- Master job dataset → Database access & load: loads
- Database schema → Neon PostgreSQL: defines
- Database access & load → Neon PostgreSQL: writes jobs
- Neon PostgreSQL → Skill extraction & classification: job descriptions
- Skill extraction & classification → Groq API: enrichment requests
- Groq API → Skill extraction & classification: skills & narrative
- Skill extraction & classification → Neon PostgreSQL: enriched skills
- Neon PostgreSQL → Market & skill analytics: job & skill data
- Market & skill analytics → Historical snapshots: current metrics
- Historical snapshots → Neon PostgreSQL: historical metrics
- Analytics views & snapshot tables → Neon PostgreSQL: defines views & tables
- Neon PostgreSQL → Streamlit analytics app: analytics datasets
- Neon PostgreSQL → Power BI dashboard: analytics model
- Neon PostgreSQL → Market report generator: market data
- Market report generator → Groq API: generation request
- Full pipeline runner → Telegram alerts: pipeline alerts