How cdngouma/skillpath-engine works
SkillPath turns job advertisements into a DuckDB labor-market warehouse for analytics and career insights. The documented workflow fetches listings and scrapes descriptions, preserves raw records, curates job metadata and canonical roles, then extracts requirement sections and uses an LLM to structure skills, certifications, experience, and seniority. SQL views support downstream analysis. The build-table implementation and SQL contents were not sampled, so those documented stages are shown without detailing their internal transformations; edges follow documented architecture or sampled calls.
Main entry points: db.py Database access · adzuna.py Adzuna ingestion · extract_requirements.py Requirement extraction command · build_jobs_table.py Job table builder
Components 18
Job ingestion
- Ingestion command
scripts/ingest.py - Adzuna ingestion
src/ingestion/adzuna.py - Description scraper
src/ingestion/scrape.py
Job curation
- Job table builder
scripts/build_jobs_table.py - Role mapping
src/processing/role_mapper.py
Requirement extraction
- Requirement extraction command
scripts/extract_requirements.py - Section extraction
src/processing/section_extractor.py - Structured extraction
src/processing/requirement_extractor.py
Warehouse and analytics
- Raw job tables
sql/schema.sql - Database access
src/storage/db.py - Curated jobs
sql/schema.sql - Job requirements
sql/schema.sql - Analytics views
sql/views.sql
Other components
- Pipeline operator
- Adzuna API
- Job posting sites
- Ollama model
- Analytics consumer
Connections 25
- Pipeline operator → Ingestion command: runs
- Ingestion command → Adzuna ingestion: dispatches
- Adzuna ingestion → Adzuna API: fetches listings
- Adzuna ingestion → Database access: stores raw listings
- Adzuna ingestion → Database access: requests missing descriptions
- Adzuna ingestion → Description scraper: scrapes descriptions
- Description scraper → Job posting sites: fetches HTML
- Description scraper → Adzuna ingestion: returns HTML
- Database access → Raw job tables: reads and writes
- Pipeline operator → Job table builder: runs
- Job table builder → Database access: reads raw metadata
- Job table builder → Role mapping: maps titles
- Job table builder → Database access: stores curated jobs
- Database access → Curated jobs: reads and writes
- Pipeline operator → Requirement extraction command: runs
- Requirement extraction command → Database access: loads job descriptions
- Requirement extraction command → Section extraction: extracts requirement text
- Requirement extraction command → Structured extraction: structures requirements
- Structured extraction → Ollama model: requests extraction
- Requirement extraction command → Database access: stores requirements
- Database access → Job requirements: reads and writes
- Analytics views → Raw job tables: queries warehouse
- Analytics views → Curated jobs: queries warehouse
- Analytics views → Job requirements: queries warehouse
- Analytics consumer → Analytics views: queries insights