How dhanush-10-1/jobpulse works
JobPulse ingests Python.org listings, validates and parses them, then upserts canonical-URL records and retains stale listings as inactive history. Its principal user flow is the React job explorer fetching through the FastAPI service and presenting searchable, paginated listings and analytics. Separate AI capabilities cover enrichment, embeddings, semantic search, matching, and resume-based recommendations; Kafka event handling and scheduled Airflow ingestion are additional paths. The graph follows shown calls and SOURCE INDEX references; several AI and messaging internals were not excerpted, so uncertain links are omitted.
Main entry points: repository.py PostgreSQL job store · pipeline.py Ingestion pipeline · App.jsx Explorer app · api.py FastAPI service
Components 28
Ingestion
- Ingestion pipeline
jobpulse/ingestion/pipeline.py - HTML fetcher
jobpulse/ingestion/fetch.py - Listing parser
jobpulse/ingestion/parser.py - Job model
jobpulse/ingestion/models.py - JSON exporter
jobpulse/storage/exporter.py
Data services
- PostgreSQL job store
jobpulse/database/repository.py - DB connection
jobpulse/database/connections.py - Configuration
config.py
AI capabilities
- Job enrichment
jobpulse/ai/enrichment.py - Job embeddings
jobpulse/ai/embeddings.py - Semantic search
jobpulse/ai/search.py - Job matching
jobpulse/ai/matching.py - Resume analysis
jobpulse/ai/resume.py - Recommendations
jobpulse/ai/recommendations.py - Gemini client
jobpulse/ai/client.py
Job explorer
- FastAPI service
api.py - Explorer app
frontend/src/App.jsx - Search filters
frontend/src/components/SearchFilters.jsx - Job cards
frontend/src/components/JobCard.jsx - Job details
frontend/src/components/JobDetails.jsx - Pagination
frontend/src/components/Pagination.jsx - Analytics dashboard
frontend/src/components/AnalyticsDashboard.jsx
Event and scheduling
- Airflow DAG
jobpulse/dags/jobpulse_ingestion.py - Kafka producer
jobpulse/messaging/producer.py - Kafka consumer
jobpulse/messaging/consumer.py
Other components
- CLI operator
- Python.org jobs
- Gemini API
Connections 30
- CLI operator → Ingestion pipeline: runs
- Ingestion pipeline → HTML fetcher: fetches HTML
jobpulse/ingestion/pipeline.py - HTML fetcher → Python.org jobs: requests listings
- Ingestion pipeline → Listing parser: parses listings
jobpulse/ingestion/pipeline.py - Listing parser → Job model: creates jobs
jobpulse/ingestion/parser.py - Ingestion pipeline → PostgreSQL job store: upserts jobs
jobpulse/ingestion/pipeline.py - Ingestion pipeline → JSON exporter: exports jobs
jobpulse/ingestion/pipeline.py - Airflow DAG → Ingestion pipeline: orchestrates
jobpulse/dags/jobpulse_ingestion.py - Kafka consumer → PostgreSQL job store: upserts events
jobpulse/messaging/consumer.py - Explorer app → Search filters: renders
frontend/src/App.jsx - Explorer app → Job cards: renders
frontend/src/App.jsx - Explorer app → Job details: renders
frontend/src/App.jsx - Explorer app → Pagination: renders
frontend/src/App.jsx - Explorer app → Analytics dashboard: renders
frontend/src/App.jsx - Explorer app → FastAPI service: requests jobs
- FastAPI service → Job matching: dispatches
api.py - FastAPI service → Semantic search: dispatches
api.py - FastAPI service → PostgreSQL job store: queries jobs
api.py - PostgreSQL job store → DB connection: connects through
jobpulse/database/repository.py - Ingestion pipeline → Configuration: reads settings
jobpulse/ingestion/pipeline.py - Job embeddings → Gemini client: requests vectors
jobpulse/ai/embeddings.py - Job embeddings → PostgreSQL job store: saves vectors
jobpulse/ai/embeddings.py - Gemini client → Gemini API: calls provider
jobpulse/ai/client.py - Job enrichment → PostgreSQL job store: reads and saves
jobpulse/ai/enrichment.py - Job enrichment → Gemini client: uses client
jobpulse/ai/enrichment.py - Job matching → PostgreSQL job store: reads jobs
jobpulse/ai/matching.py - Semantic search → PostgreSQL job store: searches vectors
jobpulse/ai/search.py - Resume analysis → Gemini client: uses client
jobpulse/ai/resume.py - Resume analysis → Gemini API: requests analysis
- Recommendations → PostgreSQL job store: uses job data