# cdngouma/skillpath-engine architecture

> Architecture diagram of the GitHub repository cdngouma/skillpath-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cdngouma/skillpath-engine
- Repository: https://github.com/cdngouma/skillpath-engine

## Overview

SkillPath turns job advertisements into a DuckDB labor-market warehouse for analytics and career insights. The documented workflow fetches listings and scrapes descriptions, preserves raw records, curates job metadata and canonical roles, then extracts requirement sections and uses an LLM to structure skills, certifications, experience, and seniority. SQL views support downstream analysis. The build-table implementation and SQL contents were not sampled, so those documented stages are shown without detailing their internal transformations; edges follow documented architecture or sampled calls.

Main entry points: [`src/storage/db.py`](https://github.com/cdngouma/skillpath-engine/blob/main/src/storage/db.py) (Database access), [`src/ingestion/adzuna.py`](https://github.com/cdngouma/skillpath-engine/blob/main/src/ingestion/adzuna.py) (Adzuna ingestion), [`scripts/extract_requirements.py`](https://github.com/cdngouma/skillpath-engine/blob/main/scripts/extract_requirements.py) (Requirement extraction command), [`scripts/build_jobs_table.py`](https://github.com/cdngouma/skillpath-engine/blob/main/scripts/build_jobs_table.py) (Job table builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Job ingestion"]
  node_ingest_cli["Ingestion command<br/>[ingest.py]"]
  node_adzuna_ingest["Adzuna ingestion<br/>[adzuna.py]"]
  node_scraper["Description scraper<br/>[scrape.py]"]
end

subgraph group_curation["Job curation"]
  node_curation_job["Job table builder"]
  node_role_mapper["Role mapping<br/>[role_mapper.py]"]
end

subgraph group_extraction["Requirement extraction"]
  node_extraction_job["Requirement extraction command"]
  node_section_extractor["Section extraction"]
  node_structured_extractor["Structured extraction"]
end

subgraph group_analytics["Warehouse and analytics"]
  node_raw_store[("Raw job tables<br/>[schema.sql]")]
  node_storage["Database access<br/>[db.py]"]
  node_jobs_table[("Curated jobs<br/>[schema.sql]")]
  node_requirements_table[("Job requirements<br/>[schema.sql]")]
  node_views["Analytics views<br/>[views.sql]"]
end

node_operator(("Pipeline operator"))
node_adzuna_api["Adzuna API"]
node_job_sites["Job posting sites"]
node_ollama["Ollama model"]
node_analyst(("Analytics consumer"))

node_operator -->|"runs"| node_ingest_cli
node_ingest_cli -->|"dispatches"| node_adzuna_ingest
node_adzuna_ingest -->|"fetches listings"| node_adzuna_api
node_adzuna_ingest -->|"stores raw listings"| node_storage
node_adzuna_ingest -->|"requests missing descriptions"| node_storage
node_adzuna_ingest -->|"scrapes descriptions"| node_scraper
node_scraper -->|"fetches HTML"| node_job_sites
node_scraper -->|"returns HTML"| node_adzuna_ingest
node_storage -->|"reads and writes"| node_raw_store
node_operator -->|"runs"| node_curation_job
node_curation_job -->|"reads raw metadata"| node_storage
node_curation_job -->|"maps titles"| node_role_mapper
node_curation_job -->|"stores curated jobs"| node_storage
node_storage -->|"reads and writes"| node_jobs_table
node_operator -->|"runs"| node_extraction_job
node_extraction_job -->|"loads job descriptions"| node_storage
node_extraction_job -->|"extracts requirement text"| node_section_extractor
node_extraction_job -->|"structures requirements"| node_structured_extractor
node_structured_extractor -->|"requests extraction"| node_ollama
node_extraction_job -->|"stores requirements"| node_storage
node_storage -->|"reads and writes"| node_requirements_table
node_views -->|"queries warehouse"| node_raw_store
node_views -->|"queries warehouse"| node_jobs_table
node_views -->|"queries warehouse"| node_requirements_table
node_analyst -->|"queries insights"| node_views

click node_ingest_cli "https://github.com/cdngouma/skillpath-engine/blob/main/scripts/ingest.py"
click node_adzuna_ingest "https://github.com/cdngouma/skillpath-engine/blob/main/src/ingestion/adzuna.py"
click node_scraper "https://github.com/cdngouma/skillpath-engine/blob/main/src/ingestion/scrape.py"
click node_raw_store "https://github.com/cdngouma/skillpath-engine/blob/main/sql/schema.sql"
click node_storage "https://github.com/cdngouma/skillpath-engine/blob/main/src/storage/db.py"
click node_curation_job "https://github.com/cdngouma/skillpath-engine/blob/main/scripts/build_jobs_table.py"
click node_role_mapper "https://github.com/cdngouma/skillpath-engine/blob/main/src/processing/role_mapper.py"
click node_jobs_table "https://github.com/cdngouma/skillpath-engine/blob/main/sql/schema.sql"
click node_extraction_job "https://github.com/cdngouma/skillpath-engine/blob/main/scripts/extract_requirements.py"
click node_section_extractor "https://github.com/cdngouma/skillpath-engine/blob/main/src/processing/section_extractor.py"
click node_structured_extractor "https://github.com/cdngouma/skillpath-engine/blob/main/src/processing/requirement_extractor.py"
click node_requirements_table "https://github.com/cdngouma/skillpath-engine/blob/main/sql/schema.sql"
click node_views "https://github.com/cdngouma/skillpath-engine/blob/main/sql/views.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_cli,node_adzuna_ingest,node_scraper toneBlue
class node_curation_job,node_role_mapper toneAmber
class node_extraction_job,node_section_extractor,node_structured_extractor,node_adzuna_api toneMint
class node_raw_store,node_storage,node_jobs_table,node_requirements_table,node_views toneRose
class node_operator,node_job_sites,node_ollama,node_analyst toneIndigo
```

## Components

### Job ingestion

- [Ingestion command](https://github.com/cdngouma/skillpath-engine/blob/main/scripts/ingest.py): `scripts/ingest.py`
- [Adzuna ingestion](https://github.com/cdngouma/skillpath-engine/blob/main/src/ingestion/adzuna.py): `src/ingestion/adzuna.py`
- [Description scraper](https://github.com/cdngouma/skillpath-engine/blob/main/src/ingestion/scrape.py): `src/ingestion/scrape.py`

### Job curation

- [Job table builder](https://github.com/cdngouma/skillpath-engine/blob/main/scripts/build_jobs_table.py): `scripts/build_jobs_table.py`
- [Role mapping](https://github.com/cdngouma/skillpath-engine/blob/main/src/processing/role_mapper.py): `src/processing/role_mapper.py`

### Requirement extraction

- [Requirement extraction command](https://github.com/cdngouma/skillpath-engine/blob/main/scripts/extract_requirements.py): `scripts/extract_requirements.py`
- [Section extraction](https://github.com/cdngouma/skillpath-engine/blob/main/src/processing/section_extractor.py): `src/processing/section_extractor.py`
- [Structured extraction](https://github.com/cdngouma/skillpath-engine/blob/main/src/processing/requirement_extractor.py): `src/processing/requirement_extractor.py`

### Warehouse and analytics

- [Raw job tables](https://github.com/cdngouma/skillpath-engine/blob/main/sql/schema.sql): `sql/schema.sql`
- [Database access](https://github.com/cdngouma/skillpath-engine/blob/main/src/storage/db.py): `src/storage/db.py`
- [Curated jobs](https://github.com/cdngouma/skillpath-engine/blob/main/sql/schema.sql): `sql/schema.sql`
- [Job requirements](https://github.com/cdngouma/skillpath-engine/blob/main/sql/schema.sql): `sql/schema.sql`
- [Analytics views](https://github.com/cdngouma/skillpath-engine/blob/main/sql/views.sql): `sql/views.sql`

### Other components

- **Pipeline operator**
- **Adzuna API**
- **Job posting sites**
- **Ollama model**
- **Analytics consumer**

## Connections

- Pipeline operator → Ingestion command: runs
- Ingestion command → Adzuna ingestion: dispatches
- Adzuna ingestion → Adzuna API: fetches listings
- Adzuna ingestion → Database access: stores raw listings
- Adzuna ingestion → Database access: requests missing descriptions
- Adzuna ingestion → Description scraper: scrapes descriptions
- Description scraper → Job posting sites: fetches HTML
- Description scraper → Adzuna ingestion: returns HTML
- Database access → Raw job tables: reads and writes
- Pipeline operator → Job table builder: runs
- Job table builder → Database access: reads raw metadata
- Job table builder → Role mapping: maps titles
- Job table builder → Database access: stores curated jobs
- Database access → Curated jobs: reads and writes
- Pipeline operator → Requirement extraction command: runs
- Requirement extraction command → Database access: loads job descriptions
- Requirement extraction command → Section extraction: extracts requirement text
- Requirement extraction command → Structured extraction: structures requirements
- Structured extraction → Ollama model: requests extraction
- Requirement extraction command → Database access: stores requirements
- Database access → Job requirements: reads and writes
- Analytics views → Raw job tables: queries warehouse
- Analytics views → Curated jobs: queries warehouse
- Analytics views → Job requirements: queries warehouse
- Analytics consumer → Analytics views: queries insights