# kendelja/job-lens architecture

> Architecture diagram of the GitHub repository kendelja/job-lens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kendelja/job-lens
- Repository: https://github.com/kendelja/job-lens
- Video tour (about a minute): https://gitdiagram.com/kendelja/job-lens/video

## Overview

JobLens aggregates Built In and Wellfound postings into a PostgreSQL-backed job dashboard. Ingestion is assembled by the runner, which selects source-specific scrapers and parsers; the shared pipeline normalizes, validates, filters for recency, deduplicates, and stores results. FastAPI exposes ingestion and retrieval, while the React dashboard fetches and filters jobs and supports saved-job interactions. The graph preserves documented source and storage relationships; some scraper and database details are documented rather than shown in sampled excerpts.

Main entry points: [`Backend/scripts/runIngestion.py`](https://github.com/kendelja/job-lens/blob/main/Backend/scripts/runIngestion.py) (Ingestion runner), [`Backend/app/ingestion/pipeline.py`](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/pipeline.py) (Processing pipeline), [`Backend/app/database/repositories.py`](https://github.com/kendelja/job-lens/blob/main/Backend/app/database/repositories.py) (Job repository), [`Backend/app/api/jobs.py`](https://github.com/kendelja/job-lens/blob/main/Backend/app/api/jobs.py) (Jobs endpoints)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Job ingestion"]
  node_runner["Ingestion runner<br/>[runIngestion.py]"]
  node_builtin_scraper["Built In scraper<br/>[BuiltInScraper.py]"]
  node_wellfound_scraper["Wellfound scraper"]
  node_builtin_parser["Built In parser<br/>[BuiltInParser.py]"]
  node_wellfound_parser["Wellfound parser<br/>[WellfoundParser.py]"]
  node_pipeline["Processing pipeline<br/>[pipeline.py]"]
  node_normalizer["Job normalizer<br/>[normalizer.py]"]
  node_validator["Job validator<br/>[validator.py]"]
  node_deduplicator["Job deduplicator<br/>[deduplicator.py]"]
end

subgraph group_persistence["Persistence"]
  node_job_model["Job model<br/>[job.py]"]
  node_repository["Job repository<br/>[repositories.py]"]
  node_connection["Database connection<br/>[connection.py]"]
  node_postgres[("PostgreSQL jobs<br/>[schema.sql]")]
end

subgraph group_api["Backend API"]
  node_api_app["FastAPI app<br/>[main.py]"]
  node_jobs_api["Jobs endpoints<br/>[jobs.py]"]
end

subgraph group_frontend["Dashboard"]
  node_react_entry["React entry<br/>[main.jsx]"]
  node_app["App shell<br/>[App.jsx]"]
  node_dashboard["Job dashboard<br/>[Dashboard.jsx]"]
  node_job_card["Job card<br/>[JobCard.jsx]"]
end

node_builtin_source(("Built In"))
node_wellfound_source(("Wellfound"))
node_browser(("User browser"))

node_runner -->|"runs"| node_pipeline
node_runner -->|"selects"| node_builtin_scraper
node_runner -->|"selects"| node_wellfound_scraper
node_runner -->|"selects"| node_builtin_parser
node_runner -->|"selects"| node_wellfound_parser
node_builtin_source -.->|"scrapes"| node_builtin_scraper
node_wellfound_source -.->|"scrapes"| node_wellfound_scraper
node_pipeline -->|"normalizes"| node_normalizer
node_pipeline -->|"validates"| node_validator
node_pipeline -->|"checks duplicates"| node_deduplicator
node_pipeline -->|"inserts jobs"| node_repository
node_normalizer -->|"creates"| node_job_model
node_validator -->|"validates"| node_job_model
node_repository -->|"opens connection"| node_connection
node_repository -->|"reads fields"| node_job_model
node_repository -.->|"reads and writes"| node_postgres
node_api_app -->|"registers router"| node_jobs_api
node_jobs_api -->|"retrieves and clears"| node_repository
node_jobs_api -->|"starts ingestion"| node_runner
node_react_entry -->|"renders"| node_app
node_app -->|"renders"| node_dashboard
node_dashboard -.->|"fetches jobs"| node_jobs_api
node_browser -.->|"uses"| node_dashboard

click node_runner "https://github.com/kendelja/job-lens/blob/main/Backend/scripts/runIngestion.py"
click node_builtin_scraper "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/scrapers/BuiltInScraper.py"
click node_wellfound_scraper "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/scrapers/WellfoundScraper.py"
click node_builtin_parser "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/parsers/BuiltInParser.py"
click node_wellfound_parser "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/parsers/WellfoundParser.py"
click node_pipeline "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/pipeline.py"
click node_normalizer "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/normalizer.py"
click node_validator "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/validator.py"
click node_deduplicator "https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/deduplicator.py"
click node_job_model "https://github.com/kendelja/job-lens/blob/main/Backend/app/models/job.py"
click node_repository "https://github.com/kendelja/job-lens/blob/main/Backend/app/database/repositories.py"
click node_connection "https://github.com/kendelja/job-lens/blob/main/Backend/app/database/connection.py"
click node_postgres "https://github.com/kendelja/job-lens/blob/main/database/schema.sql"
click node_api_app "https://github.com/kendelja/job-lens/blob/main/Backend/app/main.py"
click node_jobs_api "https://github.com/kendelja/job-lens/blob/main/Backend/app/api/jobs.py"
click node_react_entry "https://github.com/kendelja/job-lens/blob/main/frontend/src/main.jsx"
click node_app "https://github.com/kendelja/job-lens/blob/main/frontend/src/App.jsx"
click node_dashboard "https://github.com/kendelja/job-lens/blob/main/frontend/src/pages/Dashboard.jsx"
click node_job_card "https://github.com/kendelja/job-lens/blob/main/frontend/src/components/JobCard.jsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_builtin_scraper,node_wellfound_scraper,node_builtin_parser,node_wellfound_parser,node_pipeline,node_normalizer,node_validator,node_deduplicator,node_browser toneBlue
class node_job_model,node_repository,node_connection,node_postgres toneAmber
class node_api_app,node_jobs_api toneMint
class node_react_entry,node_app,node_dashboard,node_job_card toneRose
class node_builtin_source,node_wellfound_source toneIndigo
```

## Components

### Job ingestion

- [Ingestion runner](https://github.com/kendelja/job-lens/blob/main/Backend/scripts/runIngestion.py): `Backend/scripts/runIngestion.py`
- [Built In scraper](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/scrapers/BuiltInScraper.py): `Backend/app/ingestion/scrapers/BuiltInScraper.py`
- [Wellfound scraper](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/scrapers/WellfoundScraper.py): `Backend/app/ingestion/scrapers/WellfoundScraper.py`
- [Built In parser](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/parsers/BuiltInParser.py): `Backend/app/ingestion/parsers/BuiltInParser.py`
- [Wellfound parser](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/parsers/WellfoundParser.py): `Backend/app/ingestion/parsers/WellfoundParser.py`
- [Processing pipeline](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/pipeline.py): `Backend/app/ingestion/pipeline.py`
- [Job normalizer](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/normalizer.py): `Backend/app/ingestion/normalizer.py`
- [Job validator](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/validator.py): `Backend/app/ingestion/validator.py`
- [Job deduplicator](https://github.com/kendelja/job-lens/blob/main/Backend/app/ingestion/deduplicator.py): `Backend/app/ingestion/deduplicator.py`

### Persistence

- [Job model](https://github.com/kendelja/job-lens/blob/main/Backend/app/models/job.py): `Backend/app/models/job.py`
- [Job repository](https://github.com/kendelja/job-lens/blob/main/Backend/app/database/repositories.py): `Backend/app/database/repositories.py`
- [Database connection](https://github.com/kendelja/job-lens/blob/main/Backend/app/database/connection.py): `Backend/app/database/connection.py`
- [PostgreSQL jobs](https://github.com/kendelja/job-lens/blob/main/database/schema.sql): `database/schema.sql`

### Backend API

- [FastAPI app](https://github.com/kendelja/job-lens/blob/main/Backend/app/main.py): `Backend/app/main.py`
- [Jobs endpoints](https://github.com/kendelja/job-lens/blob/main/Backend/app/api/jobs.py): `Backend/app/api/jobs.py`

### Dashboard

- [React entry](https://github.com/kendelja/job-lens/blob/main/frontend/src/main.jsx): `frontend/src/main.jsx`
- [App shell](https://github.com/kendelja/job-lens/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`
- [Job dashboard](https://github.com/kendelja/job-lens/blob/main/frontend/src/pages/Dashboard.jsx): `frontend/src/pages/Dashboard.jsx`
- [Job card](https://github.com/kendelja/job-lens/blob/main/frontend/src/components/JobCard.jsx): `frontend/src/components/JobCard.jsx`

### Other components

- **Built In**
- **Wellfound**
- **User browser**

## Connections

- Ingestion runner → Processing pipeline: runs (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Built In scraper: selects (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Wellfound scraper: selects (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Built In parser: selects (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Wellfound parser: selects (evidence: `Backend/scripts/runIngestion.py`)
- Built In → Built In scraper: scrapes (evidence: `README.md`)
- Wellfound → Wellfound scraper: scrapes (evidence: `README.md`)
- Processing pipeline → Job normalizer: normalizes (evidence: `Backend/app/ingestion/pipeline.py`)
- Processing pipeline → Job validator: validates (evidence: `Backend/app/ingestion/pipeline.py`)
- Processing pipeline → Job deduplicator: checks duplicates (evidence: `Backend/app/ingestion/pipeline.py`)
- Processing pipeline → Job repository: inserts jobs (evidence: `Backend/app/ingestion/pipeline.py`)
- Job normalizer → Job model: creates (evidence: `Backend/app/ingestion/normalizer.py`)
- Job validator → Job model: validates (evidence: `Backend/app/ingestion/validator.py`)
- Job repository → Database connection: opens connection (evidence: `Backend/app/database/repositories.py`)
- Job repository → Job model: reads fields (evidence: `Backend/app/database/repositories.py`)
- Job repository → PostgreSQL jobs: reads and writes (evidence: `README.md`)
- FastAPI app → Jobs endpoints: registers router (evidence: `Backend/app/main.py`)
- Jobs endpoints → Job repository: retrieves and clears (evidence: `Backend/app/api/jobs.py`)
- Jobs endpoints → Ingestion runner: starts ingestion (evidence: `Backend/app/api/jobs.py`)
- React entry → App shell: renders (evidence: `frontend/src/main.jsx`)
- App shell → Job dashboard: renders (evidence: `frontend/src/App.jsx`)
- Job dashboard → Jobs endpoints: fetches jobs (evidence: `frontend/src/pages/Dashboard.jsx`)
- User browser → Job dashboard: uses