How vvinayy/job-market-intelligence works
Job Market Intelligence collects sampled Indian IT job postings, cleans and stores them, then serves search and market analytics through an API to a multipage dashboard. Its separate daily liveness pass revisits stored URLs to verify expiry. The README documents the principal flow and dashboard capabilities; source excerpts and the index support the shown module relationships. Hirist collection is a distinct, documented capability, while its liveness probe is explicitly not scheduled.
Main entry points: dash_common.py Dashboard API client · main.py FastAPI application · database.py API database access · job_database.py Posting storage
Components 27
Collection and processing
- Naukri scraper
services/naukri_collector.py - Hirist scraper
services/hirist_collector.py - Skill extraction
services/skill_taxonomy.py - Record cleaning
services/cleaning.py
Persistence and history
- Posting storage
database/job_database.py - Daily skill snapshots
database/trends_setup.sql
API and analytics
- FastAPI application
backend/api/main.py - API database access
backend/api/database.py - Response models
backend/api/models.py - Posting search
backend/api/routers/postings.py - Reference lookups
backend/api/routers/reference.py - Market analytics
backend/api/routers/analytics.py - Trend endpoints
backend/api/routers/trends.py
Dashboard
- Dashboard API client
frontend/dash_common.py - Dashboard home
frontend/Home.py - Skills page
frontend/pages/1_Skills.py - Market page
frontend/pages/2_Market.py - Trends page
frontend/pages/3_Trends.py - Composition page
frontend/pages/4_Composition.py - Jobs page
frontend/pages/5_Jobs.py
Liveness tracking
- URL recheck run
services/liveness_checker.py - Expiry classification
services/liveness.py - Run notifications
communication/notify.py
Other components
- Dashboard user
- Naukri.com
- Hirist
- PostgreSQL
Connections 35
- Naukri.com → Naukri scraper: serves postings
README.md - Naukri scraper → Skill extraction: extracts skills
services/naukri_collector.py - Naukri scraper → Posting storage: saves records
README.md - Naukri scraper → Daily skill snapshots: records snapshot
README.md - Skill extraction → Record cleaning: supplies extraction
services/cleaning.py - Posting storage → Record cleaning: cleans records
database/job_database.py - Hirist → Hirist scraper: serves postings
README.md - Hirist scraper → Posting storage: saves records
services/hirist_collector.py - Posting storage → PostgreSQL: writes postings
README.md - Daily skill snapshots → PostgreSQL: stores history
README.md - FastAPI application → API database access: manages pool
backend/api/main.py - FastAPI application → Posting search: registers router
backend/api/main.py - FastAPI application → Reference lookups: registers router
backend/api/main.py - FastAPI application → Market analytics: registers router
backend/api/main.py - FastAPI application → Trend endpoints: registers router
backend/api/main.py - API database access → PostgreSQL: queries
backend/api/database.py - Posting search → API database access: queries
backend/api/routers/postings.py - Reference lookups → API database access: queries
backend/api/routers/reference.py - Market analytics → API database access: queries
backend/api/routers/analytics.py - Trend endpoints → API database access: queries
backend/api/routers/trends.py - Posting search → Response models: validates responses
backend/api/routers/postings.py - Reference lookups → Response models: shapes responses
backend/api/routers/reference.py - Market analytics → Response models: shapes responses
backend/api/routers/analytics.py - Trend endpoints → Response models: shapes responses
backend/api/routers/trends.py - Dashboard user → Dashboard API client: requests views
README.md - Dashboard API client → FastAPI application: calls over HTTP
README.md - Dashboard home → Dashboard API client: uses helpers
frontend/Home.py - Skills page → Dashboard API client: uses helpers
frontend/pages/1_Skills.py - Market page → Dashboard API client: uses helpers
frontend/pages/2_Market.py - Trends page → Dashboard API client: uses helpers
frontend/pages/3_Trends.py - Composition page → Dashboard API client: uses helpers
frontend/pages/4_Composition.py - Jobs page → Dashboard API client: uses helpers
frontend/pages/5_Jobs.py - URL recheck run → Posting storage: checks stored URLs
README.md - URL recheck run → Expiry classification: classifies responses
services/liveness_checker.py - URL recheck run → Run notifications: reports outcomes
services/liveness_checker.py