# kiyst/jobgoblin architecture

> Architecture diagram of the GitHub repository kiyst/jobgoblin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kiyst/jobgoblin
- Repository: https://github.com/kiyst/jobgoblin

## Overview

JobGoblin is an early-stage job aggregation and market-intelligence backend. Its principal workflow is to plan a saved-search query for providers, execute discovery, and persist ingestion outcomes and job identity; normalization is a distinct market-data capability. The README documents the FastAPI server and health/readiness endpoints, while source-index references expose the ingestion, provider, persistence, and saved-search connections. Several planned product areas are represented by models or services but are not shown wired into the main runtime flow.

Main entry points: [`backend/app/ingestion/provider_execution.py`](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/provider_execution.py) (Provider execution), [`backend/app/ingestion/persistence.py`](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/persistence.py) (Job persistence), [`backend/app/discovery/query_planner.py`](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/discovery/query_planner.py) (Query planner), [`backend/app/providers/base.py`](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/base.py) (Provider interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["API entry"]
  node_main["FastAPI app<br/>[main.py]"]
  node_health["Health endpoints<br/>[health.py]"]
end

subgraph group_search["Search planning"]
  node_search_service["Saved searches<br/>[saved_searches.py]"]
  node_planner["Query planner<br/>[query_planner.py]"]
  node_query_contract["Query contracts<br/>[provider.py]"]
end

subgraph group_collection["Job collection"]
  node_orchestrator["Collection orchestrator<br/>[orchestrator.py]"]
  node_pipeline["Ingestion pipeline<br/>[pipeline.py]"]
  node_registry["Provider registry<br/>[registry.py]"]
  node_provider["Provider interface<br/>[base.py]"]
  node_fixture["Fixture provider<br/>[fixture.py]"]
  node_execution["Provider execution"]
  node_job_contract["Discovered job<br/>[discovered_job.py]"]
end

subgraph group_data["Domain data"]
  node_search_store[("Saved-search models<br/>[saved_search.py]")]
  node_persistence["Job persistence<br/>[persistence.py]"]
  node_identity["Identity resolution<br/>[identity.py]"]
  node_natural_key["Natural keys<br/>[natural_key.py]"]
  node_orm[("Domain models")]
  node_profiles["Candidate profiles"]
  node_user_jobs["User job actions<br/>[user_jobs.py]"]
end

subgraph group_market["Market intelligence"]
  node_normalization["Job normalization"]
  node_provenance["Normalization results<br/>[types.py]"]
end

node_client(("API client"))

node_client -->|"requests API"| node_main
node_main -->|"mounts routes"| node_health
node_search_service -->|"uses models"| node_search_store
node_planner -->|"reads search"| node_search_store
node_planner -->|"builds query"| node_query_contract
node_orchestrator -->|"plans queries"| node_planner
node_orchestrator -->|"gets providers"| node_registry
node_registry -->|"registers providers"| node_provider
node_fixture -->|"implements contract"| node_provider
node_pipeline -->|"executes query"| node_execution
node_execution -->|"calls provider"| node_provider
node_execution -->|"handles results"| node_job_contract
node_execution -->|"persists jobs"| node_persistence
node_persistence -->|"resolves identity"| node_identity
node_persistence -->|"computes keys"| node_natural_key
node_persistence -->|"writes records"| node_orm
node_profiles -->|"uses models"| node_orm
node_user_jobs -.->|"uses models"| node_orm
node_normalization -->|"returns typed results"| node_provenance

click node_main "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/main.py"
click node_health "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/api/health.py"
click node_search_service "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/services/saved_searches.py"
click node_search_store "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/db/models/saved_search.py"
click node_planner "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/discovery/query_planner.py"
click node_query_contract "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/schemas/provider.py"
click node_orchestrator "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/orchestrator.py"
click node_pipeline "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/pipeline.py"
click node_registry "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/registry.py"
click node_provider "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/base.py"
click node_fixture "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/fixture.py"
click node_execution "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/provider_execution.py"
click node_job_contract "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/schemas/discovered_job.py"
click node_persistence "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/persistence.py"
click node_identity "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/identity.py"
click node_natural_key "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/natural_key.py"
click node_orm "https://github.com/kiyst/jobgoblin/tree/codex/phase1-users-wip/backend/app/db/models"
click node_profiles "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/services/candidate_profiles.py"
click node_user_jobs "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/services/user_jobs.py"
click node_normalization "https://github.com/kiyst/jobgoblin/tree/codex/phase1-users-wip/backend/app/normalization"
click node_provenance "https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/normalization/types.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_health,node_client toneBlue
class node_search_service,node_planner,node_query_contract toneAmber
class node_orchestrator,node_pipeline,node_registry,node_provider,node_fixture,node_execution,node_job_contract toneMint
class node_search_store,node_persistence,node_identity,node_natural_key,node_orm,node_profiles,node_user_jobs toneRose
class node_normalization,node_provenance toneIndigo
```

## Components

### API entry

- [FastAPI app](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/main.py): `backend/app/main.py`
- [Health endpoints](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/api/health.py): `backend/app/api/health.py`

### Search planning

- [Saved searches](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/services/saved_searches.py): `backend/app/services/saved_searches.py`
- [Query planner](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/discovery/query_planner.py): `backend/app/discovery/query_planner.py`
- [Query contracts](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/schemas/provider.py): `backend/app/schemas/provider.py`

### Job collection

- [Collection orchestrator](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/orchestrator.py): `backend/app/ingestion/orchestrator.py`
- [Ingestion pipeline](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/pipeline.py): `backend/app/ingestion/pipeline.py`
- [Provider registry](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/registry.py): `backend/app/providers/registry.py`
- [Provider interface](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/base.py): `backend/app/providers/base.py`
- [Fixture provider](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/providers/fixture.py): `backend/app/providers/fixture.py`
- [Provider execution](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/provider_execution.py): `backend/app/ingestion/provider_execution.py`
- [Discovered job](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/schemas/discovered_job.py): `backend/app/schemas/discovered_job.py`

### Domain data

- [Saved-search models](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/db/models/saved_search.py): `backend/app/db/models/saved_search.py`
- [Job persistence](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/persistence.py): `backend/app/ingestion/persistence.py`
- [Identity resolution](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/identity.py): `backend/app/ingestion/identity.py`
- [Natural keys](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/ingestion/natural_key.py): `backend/app/ingestion/natural_key.py`
- [Domain models](https://github.com/kiyst/jobgoblin/tree/codex/phase1-users-wip/backend/app/db/models): `backend/app/db/models`
- [Candidate profiles](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/services/candidate_profiles.py): `backend/app/services/candidate_profiles.py`
- [User job actions](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/services/user_jobs.py): `backend/app/services/user_jobs.py`

### Market intelligence

- [Job normalization](https://github.com/kiyst/jobgoblin/tree/codex/phase1-users-wip/backend/app/normalization): `backend/app/normalization`
- [Normalization results](https://github.com/kiyst/jobgoblin/blob/codex/phase1-users-wip/backend/app/normalization/types.py): `backend/app/normalization/types.py`

### Other components

- **API client**

## Connections

- API client → FastAPI app: requests API (evidence: `README.md`)
- FastAPI app → Health endpoints: mounts routes (evidence: `backend/app/main.py`)
- Saved searches → Saved-search models: uses models (evidence: `backend/app/services/saved_searches.py`)
- Query planner → Saved-search models: reads search (evidence: `backend/app/discovery/query_planner.py`)
- Query planner → Query contracts: builds query (evidence: `backend/app/discovery/query_planner.py`)
- Collection orchestrator → Query planner: plans queries (evidence: `backend/app/ingestion/orchestrator.py`)
- Collection orchestrator → Provider registry: gets providers (evidence: `backend/app/ingestion/orchestrator.py`)
- Provider registry → Provider interface: registers providers (evidence: `backend/app/providers/registry.py`)
- Fixture provider → Provider interface: implements contract (evidence: `backend/app/providers/fixture.py`)
- Ingestion pipeline → Provider execution: executes query (evidence: `backend/app/ingestion/pipeline.py`)
- Provider execution → Provider interface: calls provider (evidence: `backend/app/ingestion/provider_execution.py`)
- Provider execution → Discovered job: handles results (evidence: `backend/app/ingestion/provider_execution.py`)
- Provider execution → Job persistence: persists jobs (evidence: `backend/app/ingestion/provider_execution.py`)
- Job persistence → Identity resolution: resolves identity (evidence: `backend/app/ingestion/persistence.py`)
- Job persistence → Natural keys: computes keys (evidence: `backend/app/ingestion/persistence.py`)
- Job persistence → Domain models: writes records (evidence: `backend/app/ingestion/persistence.py`)
- Candidate profiles → Domain models: uses models (evidence: `backend/app/services/candidate_profiles.py`)
- User job actions → Domain models: uses models
- Job normalization → Normalization results: returns typed results (evidence: `backend/app/normalization/types.py`)