# vvinayy/job-market-intelligence architecture

> Architecture diagram of the GitHub repository vvinayy/job-market-intelligence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vvinayy/job-market-intelligence
- Repository: https://github.com/vvinayy/job-market-intelligence

## Overview

Job Market Intelligence collects sampled Indian IT job postings, cleans and stores them, then serves search and market analytics through an API to a multipage dashboard. Its separate daily liveness pass revisits stored URLs to verify expiry. The README documents the principal flow and dashboard capabilities; source excerpts and the index support the shown module relationships. Hirist collection is a distinct, documented capability, while its liveness probe is explicitly not scheduled.

Main entry points: [`frontend/dash_common.py`](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/dash_common.py) (Dashboard API client), [`backend/api/main.py`](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/main.py) (FastAPI application), [`backend/api/database.py`](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/database.py) (API database access), [`database/job_database.py`](https://github.com/vvinayy/job-market-intelligence/blob/main/database/job_database.py) (Posting storage)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collect["Collection and processing"]
  node_naukri_collector["Naukri scraper"]
  node_hirist_collector["Hirist scraper"]
  node_skill_taxonomy["Skill extraction<br/>[skill_taxonomy.py]"]
  node_cleaning["Record cleaning<br/>[cleaning.py]"]
end

subgraph group_storage["Persistence and history"]
  node_job_database["Posting storage<br/>[job_database.py]"]
  node_trend_snapshots["Daily skill snapshots<br/>[trends_setup.sql]"]
end

subgraph group_api["API and analytics"]
  node_api_app["FastAPI application<br/>[main.py]"]
  node_api_database["API database access<br/>[database.py]"]
  node_api_models["Response models<br/>[models.py]"]
  node_postings_api["Posting search<br/>[postings.py]"]
  node_reference_api["Reference lookups<br/>[reference.py]"]
  node_analytics_api["Market analytics<br/>[analytics.py]"]
  node_trends_api["Trend endpoints<br/>[trends.py]"]
end

subgraph group_dashboard["Dashboard"]
  node_dashboard_client["Dashboard API client<br/>[dash_common.py]"]
  node_home["Dashboard home<br/>[Home.py]"]
  node_skills_page["Skills page<br/>[1_Skills.py]"]
  node_market_page["Market page<br/>[2_Market.py]"]
  node_trends_page["Trends page<br/>[3_Trends.py]"]
  node_composition_page["Composition page<br/>[4_Composition.py]"]
  node_jobs_page["Jobs page<br/>[5_Jobs.py]"]
end

subgraph group_liveness["Liveness tracking"]
  node_liveness_checker["URL recheck run"]
  node_liveness_rules["Expiry classification<br/>[liveness.py]"]
  node_notifications["Run notifications<br/>[notify.py]"]
end

node_analyst(("Dashboard user"))
node_naukri["Naukri.com"]
node_hirist["Hirist"]
node_postgres[("PostgreSQL")]

node_naukri -->|"serves postings"| node_naukri_collector
node_naukri_collector -->|"extracts skills"| node_skill_taxonomy
node_naukri_collector -->|"saves records"| node_job_database
node_naukri_collector -->|"records snapshot"| node_trend_snapshots
node_skill_taxonomy -->|"supplies extraction"| node_cleaning
node_job_database -->|"cleans records"| node_cleaning
node_hirist -.->|"serves postings"| node_hirist_collector
node_hirist_collector -->|"saves records"| node_job_database
node_job_database -->|"writes postings"| node_postgres
node_trend_snapshots -->|"stores history"| node_postgres
node_api_app -->|"manages pool"| node_api_database
node_api_app -->|"registers router"| node_postings_api
node_api_app -->|"registers router"| node_reference_api
node_api_app -->|"registers router"| node_analytics_api
node_api_app -->|"registers router"| node_trends_api
node_api_database -->|"queries"| node_postgres
node_postings_api -->|"queries"| node_api_database
node_reference_api -->|"queries"| node_api_database
node_analytics_api -->|"queries"| node_api_database
node_trends_api -->|"queries"| node_api_database
node_postings_api -->|"validates responses"| node_api_models
node_reference_api -->|"shapes responses"| node_api_models
node_analytics_api -->|"shapes responses"| node_api_models
node_trends_api -->|"shapes responses"| node_api_models
node_analyst -->|"requests views"| node_dashboard_client
node_dashboard_client -->|"calls over HTTP"| node_api_app
node_home -->|"uses helpers"| node_dashboard_client
node_skills_page -->|"uses helpers"| node_dashboard_client
node_market_page -->|"uses helpers"| node_dashboard_client
node_trends_page -->|"uses helpers"| node_dashboard_client
node_composition_page -->|"uses helpers"| node_dashboard_client
node_jobs_page -->|"uses helpers"| node_dashboard_client
node_liveness_checker -->|"checks stored URLs"| node_job_database
node_liveness_checker -->|"classifies responses"| node_liveness_rules
node_liveness_checker -->|"reports outcomes"| node_notifications

click node_naukri_collector "https://github.com/vvinayy/job-market-intelligence/blob/main/services/naukri_collector.py"
click node_hirist_collector "https://github.com/vvinayy/job-market-intelligence/blob/main/services/hirist_collector.py"
click node_skill_taxonomy "https://github.com/vvinayy/job-market-intelligence/blob/main/services/skill_taxonomy.py"
click node_cleaning "https://github.com/vvinayy/job-market-intelligence/blob/main/services/cleaning.py"
click node_job_database "https://github.com/vvinayy/job-market-intelligence/blob/main/database/job_database.py"
click node_trend_snapshots "https://github.com/vvinayy/job-market-intelligence/blob/main/database/trends_setup.sql"
click node_api_app "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/main.py"
click node_api_database "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/database.py"
click node_api_models "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/models.py"
click node_postings_api "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/postings.py"
click node_reference_api "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/reference.py"
click node_analytics_api "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/analytics.py"
click node_trends_api "https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/trends.py"
click node_dashboard_client "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/dash_common.py"
click node_home "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/Home.py"
click node_skills_page "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/1_Skills.py"
click node_market_page "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/2_Market.py"
click node_trends_page "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/3_Trends.py"
click node_composition_page "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/4_Composition.py"
click node_jobs_page "https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/5_Jobs.py"
click node_liveness_checker "https://github.com/vvinayy/job-market-intelligence/blob/main/services/liveness_checker.py"
click node_liveness_rules "https://github.com/vvinayy/job-market-intelligence/blob/main/services/liveness.py"
click node_notifications "https://github.com/vvinayy/job-market-intelligence/blob/main/communication/notify.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_naukri_collector,node_hirist_collector,node_skill_taxonomy,node_cleaning,node_analyst toneBlue
class node_job_database,node_trend_snapshots,node_postgres toneAmber
class node_api_app,node_api_database,node_api_models,node_postings_api,node_reference_api,node_analytics_api,node_trends_api toneMint
class node_dashboard_client,node_home,node_skills_page,node_market_page,node_trends_page,node_composition_page,node_jobs_page toneRose
class node_liveness_checker,node_liveness_rules,node_notifications,node_naukri,node_hirist toneIndigo
```

## Components

### Collection and processing

- [Naukri scraper](https://github.com/vvinayy/job-market-intelligence/blob/main/services/naukri_collector.py): `services/naukri_collector.py`
- [Hirist scraper](https://github.com/vvinayy/job-market-intelligence/blob/main/services/hirist_collector.py): `services/hirist_collector.py`
- [Skill extraction](https://github.com/vvinayy/job-market-intelligence/blob/main/services/skill_taxonomy.py): `services/skill_taxonomy.py`
- [Record cleaning](https://github.com/vvinayy/job-market-intelligence/blob/main/services/cleaning.py): `services/cleaning.py`

### Persistence and history

- [Posting storage](https://github.com/vvinayy/job-market-intelligence/blob/main/database/job_database.py): `database/job_database.py`
- [Daily skill snapshots](https://github.com/vvinayy/job-market-intelligence/blob/main/database/trends_setup.sql): `database/trends_setup.sql`

### API and analytics

- [FastAPI application](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/main.py): `backend/api/main.py`
- [API database access](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/database.py): `backend/api/database.py`
- [Response models](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/models.py): `backend/api/models.py`
- [Posting search](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/postings.py): `backend/api/routers/postings.py`
- [Reference lookups](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/reference.py): `backend/api/routers/reference.py`
- [Market analytics](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/analytics.py): `backend/api/routers/analytics.py`
- [Trend endpoints](https://github.com/vvinayy/job-market-intelligence/blob/main/backend/api/routers/trends.py): `backend/api/routers/trends.py`

### Dashboard

- [Dashboard API client](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/dash_common.py): `frontend/dash_common.py`
- [Dashboard home](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/Home.py): `frontend/Home.py`
- [Skills page](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/1_Skills.py): `frontend/pages/1_Skills.py`
- [Market page](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/2_Market.py): `frontend/pages/2_Market.py`
- [Trends page](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/3_Trends.py): `frontend/pages/3_Trends.py`
- [Composition page](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/4_Composition.py): `frontend/pages/4_Composition.py`
- [Jobs page](https://github.com/vvinayy/job-market-intelligence/blob/main/frontend/pages/5_Jobs.py): `frontend/pages/5_Jobs.py`

### Liveness tracking

- [URL recheck run](https://github.com/vvinayy/job-market-intelligence/blob/main/services/liveness_checker.py): `services/liveness_checker.py`
- [Expiry classification](https://github.com/vvinayy/job-market-intelligence/blob/main/services/liveness.py): `services/liveness.py`
- [Run notifications](https://github.com/vvinayy/job-market-intelligence/blob/main/communication/notify.py): `communication/notify.py`

### Other components

- **Dashboard user**
- **Naukri.com**
- **Hirist**
- **PostgreSQL**

## Connections

- Naukri.com → Naukri scraper: serves postings (evidence: `README.md`)
- Naukri scraper → Skill extraction: extracts skills (evidence: `services/naukri_collector.py`)
- Naukri scraper → Posting storage: saves records (evidence: `README.md`)
- Naukri scraper → Daily skill snapshots: records snapshot (evidence: `README.md`)
- Skill extraction → Record cleaning: supplies extraction (evidence: `services/cleaning.py`)
- Posting storage → Record cleaning: cleans records (evidence: `database/job_database.py`)
- Hirist → Hirist scraper: serves postings (evidence: `README.md`)
- Hirist scraper → Posting storage: saves records (evidence: `services/hirist_collector.py`)
- Posting storage → PostgreSQL: writes postings (evidence: `README.md`)
- Daily skill snapshots → PostgreSQL: stores history (evidence: `README.md`)
- FastAPI application → API database access: manages pool (evidence: `backend/api/main.py`)
- FastAPI application → Posting search: registers router (evidence: `backend/api/main.py`)
- FastAPI application → Reference lookups: registers router (evidence: `backend/api/main.py`)
- FastAPI application → Market analytics: registers router (evidence: `backend/api/main.py`)
- FastAPI application → Trend endpoints: registers router (evidence: `backend/api/main.py`)
- API database access → PostgreSQL: queries (evidence: `backend/api/database.py`)
- Posting search → API database access: queries (evidence: `backend/api/routers/postings.py`)
- Reference lookups → API database access: queries (evidence: `backend/api/routers/reference.py`)
- Market analytics → API database access: queries (evidence: `backend/api/routers/analytics.py`)
- Trend endpoints → API database access: queries (evidence: `backend/api/routers/trends.py`)
- Posting search → Response models: validates responses (evidence: `backend/api/routers/postings.py`)
- Reference lookups → Response models: shapes responses (evidence: `backend/api/routers/reference.py`)
- Market analytics → Response models: shapes responses (evidence: `backend/api/routers/analytics.py`)
- Trend endpoints → Response models: shapes responses (evidence: `backend/api/routers/trends.py`)
- Dashboard user → Dashboard API client: requests views (evidence: `README.md`)
- Dashboard API client → FastAPI application: calls over HTTP (evidence: `README.md`)
- Dashboard home → Dashboard API client: uses helpers (evidence: `frontend/Home.py`)
- Skills page → Dashboard API client: uses helpers (evidence: `frontend/pages/1_Skills.py`)
- Market page → Dashboard API client: uses helpers (evidence: `frontend/pages/2_Market.py`)
- Trends page → Dashboard API client: uses helpers (evidence: `frontend/pages/3_Trends.py`)
- Composition page → Dashboard API client: uses helpers (evidence: `frontend/pages/4_Composition.py`)
- Jobs page → Dashboard API client: uses helpers (evidence: `frontend/pages/5_Jobs.py`)
- URL recheck run → Posting storage: checks stored URLs (evidence: `README.md`)
- URL recheck run → Expiry classification: classifies responses (evidence: `services/liveness_checker.py`)
- URL recheck run → Run notifications: reports outcomes (evidence: `services/liveness_checker.py`)
