How kendelja/job-lens works
JobLens aggregates Built In and Wellfound postings into a PostgreSQL-backed job dashboard. Ingestion is assembled by the runner, which selects source-specific scrapers and parsers; the shared pipeline normalizes, validates, filters for recency, deduplicates, and stores results. FastAPI exposes ingestion and retrieval, while the React dashboard fetches and filters jobs and supports saved-job interactions. The graph preserves documented source and storage relationships; some scraper and database details are documented rather than shown in sampled excerpts.
Main entry points: runIngestion.py Ingestion runner · pipeline.py Processing pipeline · repositories.py Job repository · jobs.py Jobs endpoints
Components 22
Job ingestion
- Ingestion runner
Backend/scripts/runIngestion.py - Built In scraper
Backend/app/ingestion/scrapers/BuiltInScraper.py - Wellfound scraper
Backend/app/ingestion/scrapers/WellfoundScraper.py - Built In parser
Backend/app/ingestion/parsers/BuiltInParser.py - Wellfound parser
Backend/app/ingestion/parsers/WellfoundParser.py - Processing pipeline
Backend/app/ingestion/pipeline.py - Job normalizer
Backend/app/ingestion/normalizer.py - Job validator
Backend/app/ingestion/validator.py - Job deduplicator
Backend/app/ingestion/deduplicator.py
Persistence
- Job model
Backend/app/models/job.py - Job repository
Backend/app/database/repositories.py - Database connection
Backend/app/database/connection.py - PostgreSQL jobs
database/schema.sql
Backend API
- FastAPI app
Backend/app/main.py - Jobs endpoints
Backend/app/api/jobs.py
Dashboard
- React entry
frontend/src/main.jsx - App shell
frontend/src/App.jsx - Job dashboard
frontend/src/pages/Dashboard.jsx - Job card
frontend/src/components/JobCard.jsx
Other components
- Built In
- Wellfound
- User browser
Connections 23
- Ingestion runner → Processing pipeline: runs
Backend/scripts/runIngestion.py - Ingestion runner → Built In scraper: selects
Backend/scripts/runIngestion.py - Ingestion runner → Wellfound scraper: selects
Backend/scripts/runIngestion.py - Ingestion runner → Built In parser: selects
Backend/scripts/runIngestion.py - Ingestion runner → Wellfound parser: selects
Backend/scripts/runIngestion.py - Built In → Built In scraper: scrapes
README.md - Wellfound → Wellfound scraper: scrapes
README.md - Processing pipeline → Job normalizer: normalizes
Backend/app/ingestion/pipeline.py - Processing pipeline → Job validator: validates
Backend/app/ingestion/pipeline.py - Processing pipeline → Job deduplicator: checks duplicates
Backend/app/ingestion/pipeline.py - Processing pipeline → Job repository: inserts jobs
Backend/app/ingestion/pipeline.py - Job normalizer → Job model: creates
Backend/app/ingestion/normalizer.py - Job validator → Job model: validates
Backend/app/ingestion/validator.py - Job repository → Database connection: opens connection
Backend/app/database/repositories.py - Job repository → Job model: reads fields
Backend/app/database/repositories.py - Job repository → PostgreSQL jobs: reads and writes
README.md - FastAPI app → Jobs endpoints: registers router
Backend/app/main.py - Jobs endpoints → Job repository: retrieves and clears
Backend/app/api/jobs.py - Jobs endpoints → Ingestion runner: starts ingestion
Backend/app/api/jobs.py - React entry → App shell: renders
frontend/src/main.jsx - App shell → Job dashboard: renders
frontend/src/App.jsx - Job dashboard → Jobs endpoints: fetches jobs
frontend/src/pages/Dashboard.jsx - User browser → Job dashboard: uses