# kendelja/engineerd architecture

> Architecture diagram of the GitHub repository kendelja/engineerd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kendelja/engineerd
- Repository: https://github.com/kendelja/engineerd
- Video tour (about a minute): https://gitdiagram.com/kendelja/engineerd/video

## Overview

Engineerd aggregates Built In and Wellfound postings, processes them through a shared ingestion pipeline, stores them in PostgreSQL, and serves job listings to a React dashboard. The graph preserves source-specific scraping/parsing, validation and deduplication, API retrieval and ingestion, and dashboard search, filtering, saving, and market selection. Edges follow shown calls or README-documented capabilities; unsampled scraper, validator, deduplicator, and UI details limit some implementation-level connections.

Main entry points: [`Backend/scripts/runIngestion.py`](https://github.com/kendelja/engineerd/blob/main/Backend/scripts/runIngestion.py) (Ingestion runner), [`Backend/app/ingestion/pipeline.py`](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/pipeline.py) (Shared pipeline), [`Backend/app/database/repositories.py`](https://github.com/kendelja/engineerd/blob/main/Backend/app/database/repositories.py) (Job repository), [`Backend/app/api/jobs.py`](https://github.com/kendelja/engineerd/blob/main/Backend/app/api/jobs.py) (Jobs API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Job ingestion"]
  node_builtin_scraper["Built In scraper<br/>[BuiltInScraper.py]"]
  node_wellfound_scraper["Wellfound scraper"]
  node_builtin_parser["Built In parser<br/>[BuiltInParser.py]"]
  node_wellfound_parser["Wellfound parser<br/>[WellfoundParser.py]"]
  node_runner["Ingestion runner<br/>[runIngestion.py]"]
  node_pipeline["Shared pipeline<br/>[pipeline.py]"]
  node_normalizer["Normalizer<br/>[normalizer.py]"]
  node_validator["Validator<br/>[validator.py]"]
  node_deduplicator["Deduplicator<br/>[deduplicator.py]"]
end

subgraph group_service["API and storage"]
  node_job_model["Job model<br/>[job.py]"]
  node_repository["Job repository<br/>[repositories.py]"]
  node_connection["DB connection<br/>[connection.py]"]
  node_postgres[("PostgreSQL<br/>[schema.sql]")]
  node_jobs_api["Jobs API<br/>[jobs.py]"]
  node_app["FastAPI app<br/>[main.py]"]
end

subgraph group_frontend["Dashboard"]
  node_bootstrap["React bootstrap<br/>[main.jsx]"]
  node_app_ui["App shell<br/>[App.jsx]"]
  node_dashboard["Job dashboard<br/>[Dashboard.jsx]"]
end

node_builtin(("Built In"))
node_wellfound(("Wellfound"))
node_browser(("Dashboard user"))

node_builtin -->|"provides listings"| node_builtin_scraper
node_wellfound -->|"provides listings"| node_wellfound_scraper
node_builtin_scraper -.->|"supplies HTML"| node_builtin_parser
node_wellfound_scraper -.->|"supplies HTML"| node_wellfound_parser
node_runner -->|"configures source"| node_builtin_scraper
node_runner -->|"configures parser"| node_builtin_parser
node_runner -->|"configures source"| node_wellfound_scraper
node_runner -->|"configures parser"| node_wellfound_parser
node_runner -->|"runs twice"| node_pipeline
node_pipeline -->|"normalizes jobs"| node_normalizer
node_pipeline -->|"validates jobs"| node_validator
node_pipeline -->|"checks duplicates"| node_deduplicator
node_pipeline -->|"inserts jobs"| node_repository
node_normalizer -->|"creates model"| node_job_model
node_validator -->|"validates model"| node_job_model
node_repository -->|"reads fields"| node_job_model
node_repository -->|"opens connection"| node_connection
node_repository -->|"reads and writes"| node_postgres
node_jobs_api -->|"gets and clears jobs"| node_repository
node_jobs_api -->|"starts ingestion"| node_runner
node_app -->|"mounts router"| node_jobs_api
node_browser -.->|"opens dashboard"| node_bootstrap
node_bootstrap -->|"renders"| node_app_ui
node_app_ui -->|"renders"| node_dashboard
node_dashboard -->|"requests listings"| node_jobs_api
node_jobs_api -->|"returns listings"| node_dashboard

click node_builtin_scraper "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/scrapers/BuiltInScraper.py"
click node_wellfound_scraper "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/scrapers/WellfoundScraper.py"
click node_builtin_parser "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/parsers/BuiltInParser.py"
click node_wellfound_parser "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/parsers/WellfoundParser.py"
click node_runner "https://github.com/kendelja/engineerd/blob/main/Backend/scripts/runIngestion.py"
click node_pipeline "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/pipeline.py"
click node_normalizer "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/normalizer.py"
click node_validator "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/validator.py"
click node_deduplicator "https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/deduplicator.py"
click node_job_model "https://github.com/kendelja/engineerd/blob/main/Backend/app/models/job.py"
click node_repository "https://github.com/kendelja/engineerd/blob/main/Backend/app/database/repositories.py"
click node_connection "https://github.com/kendelja/engineerd/blob/main/Backend/app/database/connection.py"
click node_postgres "https://github.com/kendelja/engineerd/blob/main/database/schema.sql"
click node_jobs_api "https://github.com/kendelja/engineerd/blob/main/Backend/app/api/jobs.py"
click node_app "https://github.com/kendelja/engineerd/blob/main/Backend/app/main.py"
click node_bootstrap "https://github.com/kendelja/engineerd/blob/main/frontend/src/main.jsx"
click node_app_ui "https://github.com/kendelja/engineerd/blob/main/frontend/src/App.jsx"
click node_dashboard "https://github.com/kendelja/engineerd/blob/main/frontend/src/pages/Dashboard.jsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_builtin_scraper,node_wellfound_scraper,node_builtin_parser,node_wellfound_parser,node_runner,node_pipeline,node_normalizer,node_validator,node_deduplicator,node_browser toneBlue
class node_job_model,node_repository,node_connection,node_postgres,node_jobs_api,node_app toneAmber
class node_bootstrap,node_app_ui,node_dashboard toneMint
class node_builtin,node_wellfound toneIndigo
```

## Components

### Job ingestion

- [Built In scraper](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/scrapers/BuiltInScraper.py): `Backend/app/ingestion/scrapers/BuiltInScraper.py`
- [Wellfound scraper](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/scrapers/WellfoundScraper.py): `Backend/app/ingestion/scrapers/WellfoundScraper.py`
- [Built In parser](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/parsers/BuiltInParser.py): `Backend/app/ingestion/parsers/BuiltInParser.py`
- [Wellfound parser](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/parsers/WellfoundParser.py): `Backend/app/ingestion/parsers/WellfoundParser.py`
- [Ingestion runner](https://github.com/kendelja/engineerd/blob/main/Backend/scripts/runIngestion.py): `Backend/scripts/runIngestion.py`
- [Shared pipeline](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/pipeline.py): `Backend/app/ingestion/pipeline.py`
- [Normalizer](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/normalizer.py): `Backend/app/ingestion/normalizer.py`
- [Validator](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/validator.py): `Backend/app/ingestion/validator.py`
- [Deduplicator](https://github.com/kendelja/engineerd/blob/main/Backend/app/ingestion/deduplicator.py): `Backend/app/ingestion/deduplicator.py`

### API and storage

- [Job model](https://github.com/kendelja/engineerd/blob/main/Backend/app/models/job.py): `Backend/app/models/job.py`
- [Job repository](https://github.com/kendelja/engineerd/blob/main/Backend/app/database/repositories.py): `Backend/app/database/repositories.py`
- [DB connection](https://github.com/kendelja/engineerd/blob/main/Backend/app/database/connection.py): `Backend/app/database/connection.py`
- [PostgreSQL](https://github.com/kendelja/engineerd/blob/main/database/schema.sql): `database/schema.sql`
- [Jobs API](https://github.com/kendelja/engineerd/blob/main/Backend/app/api/jobs.py): `Backend/app/api/jobs.py`
- [FastAPI app](https://github.com/kendelja/engineerd/blob/main/Backend/app/main.py): `Backend/app/main.py`

### Dashboard

- [React bootstrap](https://github.com/kendelja/engineerd/blob/main/frontend/src/main.jsx): `frontend/src/main.jsx`
- [App shell](https://github.com/kendelja/engineerd/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`
- [Job dashboard](https://github.com/kendelja/engineerd/blob/main/frontend/src/pages/Dashboard.jsx): `frontend/src/pages/Dashboard.jsx`

### Other components

- **Built In**
- **Wellfound**
- **Dashboard user**

## Connections

- Built In → Built In scraper: provides listings (evidence: `README.md`)
- Wellfound → Wellfound scraper: provides listings (evidence: `README.md`)
- Built In scraper → Built In parser: supplies HTML
- Wellfound scraper → Wellfound parser: supplies HTML
- Ingestion runner → Built In scraper: configures source (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Built In parser: configures parser (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Wellfound scraper: configures source (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Wellfound parser: configures parser (evidence: `Backend/scripts/runIngestion.py`)
- Ingestion runner → Shared pipeline: runs twice (evidence: `Backend/scripts/runIngestion.py`)
- Shared pipeline → Normalizer: normalizes jobs (evidence: `Backend/app/ingestion/pipeline.py`)
- Shared pipeline → Validator: validates jobs (evidence: `Backend/app/ingestion/pipeline.py`)
- Shared pipeline → Deduplicator: checks duplicates (evidence: `Backend/app/ingestion/pipeline.py`)
- Shared pipeline → Job repository: inserts jobs (evidence: `Backend/app/ingestion/pipeline.py`)
- Normalizer → Job model: creates model (evidence: `Backend/app/ingestion/normalizer.py`)
- Validator → Job model: validates model (evidence: `Backend/app/ingestion/validator.py`)
- Job repository → Job model: reads fields (evidence: `Backend/app/database/repositories.py`)
- Job repository → DB connection: opens connection (evidence: `Backend/app/database/repositories.py`)
- Job repository → PostgreSQL: reads and writes (evidence: `Backend/app/database/repositories.py`)
- Jobs API → Job repository: gets and clears jobs (evidence: `Backend/app/api/jobs.py`)
- Jobs API → Ingestion runner: starts ingestion (evidence: `Backend/app/api/jobs.py`)
- FastAPI app → Jobs API: mounts router (evidence: `Backend/app/main.py`)
- Dashboard user → React bootstrap: opens dashboard
- React bootstrap → App shell: renders (evidence: `frontend/src/main.jsx`)
- App shell → Job dashboard: renders (evidence: `frontend/src/App.jsx`)
- Job dashboard → Jobs API: requests listings (evidence: `frontend/src/pages/Dashboard.jsx`)
- Jobs API → Job dashboard: returns listings (evidence: `Backend/app/api/jobs.py`)
