# sonphattrandeveloper/vietnam-it-job-trends architecture

> Architecture diagram of the GitHub repository sonphattrandeveloper/vietnam-it-job-trends, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sonphattrandeveloper/vietnam-it-job-trends
- Repository: https://github.com/sonphattrandeveloper/vietnam-it-job-trends

## Overview

This repository collects recent Vietnamese IT job postings and prepares a published dataset for market analysis. The main workflow crawls five job boards, deduplicates listings, fetches full descriptions, filters non-IT jobs, consolidates records, extracts structured fields with Claude, and classifies four dimensions with Jev. The README documents the stage ordering; sampled sources show crawler behavior and several service integrations. Intermediate data files are not in the tree, so they are represented by the existing data directory rather than invented file paths.

Main entry points: [`crawlers/base.py`](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/base.py) (Crawl helpers), [`crawlers/itviec.py`](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/itviec.py) (ITviec crawler), [`crawlers/topdev.py`](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/topdev.py) (TopDev crawler), [`crawlers/topcv.py`](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/topcv.py) (TopCV crawler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Job collection"]
  node_itviec["ITviec crawler<br/>[itviec.py]"]
  node_topdev["TopDev crawler<br/>[topdev.py]"]
  node_vietnamworks["VietnamWorks crawler<br/>[vietnamworks.py]"]
  node_topcv["TopCV crawler<br/>[topcv.py]"]
  node_linkedin["LinkedIn crawler<br/>[linkedin.py]"]
  node_browser["Browser helpers<br/>[browser.py]"]
  node_crawl_helpers["Crawl helpers<br/>[base.py]"]
  node_raw_data["Raw listings"]
end

subgraph group_preparation["Data preparation"]
  node_dedupe["Deduplicate<br/>[step-02-dedupe.py]"]
  node_detail_fetch["Fetch details"]
  node_full_data["Detailed listings"]
  node_tech_filter["IT filter"]
  node_consolidate["Consolidate"]
  node_consolidated_data["Consolidated listings"]
end

subgraph group_enrichment["AI enrichment"]
  node_llm_extract["Extract fields"]
  node_jev_classify["Classify categories"]
end

subgraph group_published["Published dataset"]
  node_published_data["Classified dataset"]
end

node_analyst(("Researcher"))
node_job_boards{{"Job boards"}}
node_jev_service{{"Typesafe Jev"}}
node_claude_service{{"Anthropic Claude"}}

node_analyst -->|"starts crawl"| node_itviec
node_analyst -->|"starts crawl"| node_topdev
node_analyst -->|"starts crawl"| node_vietnamworks
node_analyst -->|"starts crawl"| node_topcv
node_analyst -->|"starts crawl"| node_linkedin
node_job_boards -->|"provides listings"| node_itviec
node_job_boards -->|"provides listings"| node_topdev
node_job_boards -->|"provides listings"| node_vietnamworks
node_job_boards -->|"provides listings"| node_topcv
node_job_boards -->|"provides listings"| node_linkedin
node_itviec -->|"uses"| node_browser
node_topdev -->|"uses"| node_browser
node_topcv -->|"uses"| node_browser
node_linkedin -->|"uses"| node_browser
node_itviec -->|"writes listings"| node_crawl_helpers
node_topdev -->|"writes listings"| node_crawl_helpers
node_vietnamworks -->|"writes listings"| node_crawl_helpers
node_topcv -->|"scrapes listings"| node_crawl_helpers
node_linkedin -->|"scrapes listings"| node_crawl_helpers
node_crawl_helpers -->|"writes records"| node_raw_data
node_raw_data -->|"feeds"| node_dedupe
node_dedupe -->|"feeds"| node_detail_fetch
node_detail_fetch -->|"writes details"| node_full_data
node_full_data -->|"feeds"| node_tech_filter
node_tech_filter -->|"classifies IT"| node_jev_service
node_tech_filter -->|"feeds kept jobs"| node_consolidate
node_consolidate -->|"writes records"| node_consolidated_data
node_consolidated_data -->|"feeds"| node_llm_extract
node_llm_extract -->|"requests extraction"| node_claude_service
node_llm_extract -->|"feeds"| node_jev_classify
node_jev_classify -->|"requests categories"| node_jev_service
node_jev_classify -->|"writes dataset"| node_published_data

click node_itviec "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/itviec.py"
click node_topdev "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/topdev.py"
click node_vietnamworks "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/vietnamworks.py"
click node_topcv "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/topcv.py"
click node_linkedin "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/linkedin.py"
click node_browser "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/browser.py"
click node_crawl_helpers "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/base.py"
click node_raw_data "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/tree/main/data"
click node_dedupe "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-02-dedupe.py"
click node_detail_fetch "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-03-fetch-details.py"
click node_full_data "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/tree/main/data"
click node_tech_filter "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-04-filter-tech.py"
click node_consolidate "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-05-consolidate.py"
click node_consolidated_data "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/tree/main/data"
click node_llm_extract "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-06-extract-with-llm.py"
click node_jev_classify "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-07-extract-with-jev.py"
click node_published_data "https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/data/step-07-jev-extracted.jsonl"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_itviec,node_topdev,node_vietnamworks,node_topcv,node_linkedin,node_browser,node_crawl_helpers,node_raw_data toneBlue
class node_dedupe,node_detail_fetch,node_full_data,node_tech_filter,node_consolidate,node_consolidated_data toneAmber
class node_llm_extract,node_jev_classify toneMint
class node_published_data toneRose
class node_analyst,node_job_boards,node_jev_service,node_claude_service toneIndigo
```

## Components

### Job collection

- [ITviec crawler](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/itviec.py): `crawlers/itviec.py`
- [TopDev crawler](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/topdev.py): `crawlers/topdev.py`
- [VietnamWorks crawler](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/vietnamworks.py): `crawlers/vietnamworks.py`
- [TopCV crawler](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/topcv.py): `crawlers/topcv.py`
- [LinkedIn crawler](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/linkedin.py): `crawlers/linkedin.py`
- [Browser helpers](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/browser.py): `crawlers/browser.py`
- [Crawl helpers](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/crawlers/base.py): `crawlers/base.py`
- [Raw listings](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/tree/main/data): `data`

### Data preparation

- [Deduplicate](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-02-dedupe.py): `step-02-dedupe.py`
- [Fetch details](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-03-fetch-details.py): `step-03-fetch-details.py`
- [Detailed listings](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/tree/main/data): `data`
- [IT filter](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-04-filter-tech.py): `step-04-filter-tech.py`
- [Consolidate](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-05-consolidate.py): `step-05-consolidate.py`
- [Consolidated listings](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/tree/main/data): `data`

### AI enrichment

- [Extract fields](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-06-extract-with-llm.py): `step-06-extract-with-llm.py`
- [Classify categories](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/step-07-extract-with-jev.py): `step-07-extract-with-jev.py`

### Published dataset

- [Classified dataset](https://github.com/sonphattrandeveloper/vietnam-it-job-trends/blob/main/data/step-07-jev-extracted.jsonl): `data/step-07-jev-extracted.jsonl`

### Other components

- **Researcher**
- **Job boards**
- **Typesafe Jev**
- **Anthropic Claude**

## Connections

- Researcher → ITviec crawler: starts crawl (evidence: `README.md`)
- Researcher → TopDev crawler: starts crawl (evidence: `README.md`)
- Researcher → VietnamWorks crawler: starts crawl (evidence: `README.md`)
- Researcher → TopCV crawler: starts crawl (evidence: `README.md`)
- Researcher → LinkedIn crawler: starts crawl (evidence: `README.md`)
- Job boards → ITviec crawler: provides listings (evidence: `crawlers/itviec.py`)
- Job boards → TopDev crawler: provides listings (evidence: `crawlers/topdev.py`)
- Job boards → VietnamWorks crawler: provides listings (evidence: `crawlers/vietnamworks.py`)
- Job boards → TopCV crawler: provides listings (evidence: `crawlers/topcv.py`)
- Job boards → LinkedIn crawler: provides listings (evidence: `crawlers/linkedin.py`)
- ITviec crawler → Browser helpers: uses (evidence: `crawlers/itviec.py`)
- TopDev crawler → Browser helpers: uses (evidence: `crawlers/topdev.py`)
- TopCV crawler → Browser helpers: uses (evidence: `crawlers/topcv.py`)
- LinkedIn crawler → Browser helpers: uses (evidence: `crawlers/linkedin.py`)
- ITviec crawler → Crawl helpers: writes listings (evidence: `crawlers/itviec.py`)
- TopDev crawler → Crawl helpers: writes listings (evidence: `crawlers/topdev.py`)
- VietnamWorks crawler → Crawl helpers: writes listings (evidence: `crawlers/vietnamworks.py`)
- TopCV crawler → Crawl helpers: scrapes listings (evidence: `crawlers/topcv.py`)
- LinkedIn crawler → Crawl helpers: scrapes listings (evidence: `crawlers/linkedin.py`)
- Crawl helpers → Raw listings: writes records (evidence: `crawlers/base.py`)
- Raw listings → Deduplicate: feeds (evidence: `README.md`)
- Deduplicate → Fetch details: feeds (evidence: `README.md`)
- Fetch details → Detailed listings: writes details (evidence: `README.md`)
- Detailed listings → IT filter: feeds (evidence: `README.md`)
- IT filter → Typesafe Jev: classifies IT (evidence: `step-04-filter-tech.py`)
- IT filter → Consolidate: feeds kept jobs (evidence: `README.md`)
- Consolidate → Consolidated listings: writes records (evidence: `README.md`)
- Consolidated listings → Extract fields: feeds (evidence: `step-06-extract-with-llm.py`)
- Extract fields → Anthropic Claude: requests extraction (evidence: `step-06-extract-with-llm.py`)
- Extract fields → Classify categories: feeds (evidence: `README.md`)
- Classify categories → Typesafe Jev: requests categories (evidence: `step-07-extract-with-jev.py`)
- Classify categories → Classified dataset: writes dataset (evidence: `README.md`)
