How sonphattrandeveloper/vietnam-it-job-trends works
This repository collects recent Vietnamese IT job postings and prepares a published dataset for market analysis. The main workflow crawls five job boards, deduplicates listings, fetches full descriptions, filters non-IT jobs, consolidates records, extracts structured fields with Claude, and classifies four dimensions with Jev. The README documents the stage ordering; sampled sources show crawler behavior and several service integrations. Intermediate data files are not in the tree, so they are represented by the existing data directory rather than invented file paths.
Main entry points: base.py Crawl helpers · itviec.py ITviec crawler · topdev.py TopDev crawler · topcv.py TopCV crawler
Components 21
Job collection
- ITviec crawler
crawlers/itviec.py - TopDev crawler
crawlers/topdev.py - VietnamWorks crawler
crawlers/vietnamworks.py - TopCV crawler
crawlers/topcv.py - LinkedIn crawler
crawlers/linkedin.py - Browser helpers
crawlers/browser.py - Crawl helpers
crawlers/base.py - Raw listings
data
Data preparation
- Deduplicate
step-02-dedupe.py - Fetch details
step-03-fetch-details.py - Detailed listings
data - IT filter
step-04-filter-tech.py - Consolidate
step-05-consolidate.py - Consolidated listings
data
AI enrichment
- Extract fields
step-06-extract-with-llm.py - Classify categories
step-07-extract-with-jev.py
Published dataset
- Classified dataset
data/step-07-jev-extracted.jsonl
Other components
- Researcher
- Job boards
- Typesafe Jev
- Anthropic Claude
Connections 32
- Researcher → ITviec crawler: starts crawl
README.md - Researcher → TopDev crawler: starts crawl
README.md - Researcher → VietnamWorks crawler: starts crawl
README.md - Researcher → TopCV crawler: starts crawl
README.md - Researcher → LinkedIn crawler: starts crawl
README.md - Job boards → ITviec crawler: provides listings
crawlers/itviec.py - Job boards → TopDev crawler: provides listings
crawlers/topdev.py - Job boards → VietnamWorks crawler: provides listings
crawlers/vietnamworks.py - Job boards → TopCV crawler: provides listings
crawlers/topcv.py - Job boards → LinkedIn crawler: provides listings
crawlers/linkedin.py - ITviec crawler → Browser helpers: uses
crawlers/itviec.py - TopDev crawler → Browser helpers: uses
crawlers/topdev.py - TopCV crawler → Browser helpers: uses
crawlers/topcv.py - LinkedIn crawler → Browser helpers: uses
crawlers/linkedin.py - ITviec crawler → Crawl helpers: writes listings
crawlers/itviec.py - TopDev crawler → Crawl helpers: writes listings
crawlers/topdev.py - VietnamWorks crawler → Crawl helpers: writes listings
crawlers/vietnamworks.py - TopCV crawler → Crawl helpers: scrapes listings
crawlers/topcv.py - LinkedIn crawler → Crawl helpers: scrapes listings
crawlers/linkedin.py - Crawl helpers → Raw listings: writes records
crawlers/base.py - Raw listings → Deduplicate: feeds
README.md - Deduplicate → Fetch details: feeds
README.md - Fetch details → Detailed listings: writes details
README.md - Detailed listings → IT filter: feeds
README.md - IT filter → Typesafe Jev: classifies IT
step-04-filter-tech.py - IT filter → Consolidate: feeds kept jobs
README.md - Consolidate → Consolidated listings: writes records
README.md - Consolidated listings → Extract fields: feeds
step-06-extract-with-llm.py - Extract fields → Anthropic Claude: requests extraction
step-06-extract-with-llm.py - Extract fields → Classify categories: feeds
README.md - Classify categories → Typesafe Jev: requests categories
step-07-extract-with-jev.py - Classify categories → Classified dataset: writes dataset
README.md