# tranletrungkien1705/us-ai-jobs-crawler architecture

> Architecture diagram of the GitHub repository tranletrungkien1705/us-ai-jobs-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tranletrungkien1705/us-ai-jobs-crawler
- Repository: https://github.com/tranletrungkien1705/us-ai-jobs-crawler

## Overview

This repository runs two job-discovery workflows: a US-focused crawler that tags AI-automatable roles and upserts them to Supabase, and a Vietnam .NET job collector that enriches and accumulates listings for job boards. The README documents the US sources and daily Actions schedule; sampled code shows additional sources and the VN collection stages. The board pages and some crawler details are represented from the file tree or documentation, not inspected source excerpts.

Main entry points: [`crawl.py`](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py) (US crawler), [`vn_crawl.py`](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_crawl.py) (VN crawler), [`vn_scrape.js`](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_scrape.js) (Browser scraper), [`docs/vn.html`](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/vn.html) (Vietnam job board)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_us["US AI Jobs"]
  node_us_crawler["US crawler<br/>[crawl.py]"]
  node_us_feeds["US job feeds<br/>[crawl.py]"]
  node_us_filter["Role tagging<br/>[crawl.py]"]
end

subgraph group_vn["Vietnam .NET Jobs"]
  node_vn_crawler["VN crawler<br/>[vn_crawl.py]"]
  node_vn_linkedin["LinkedIn jobs"]
  node_itviec["ITviec jobs"]
  node_remote_feeds["Remote feeds"]
  node_vn_enrich["Fit enrichment<br/>[vn_crawl.py]"]
  node_browser_scraper["Browser scraper<br/>[vn_scrape.js]"]
  node_topcv["TopCV jobs"]
  node_vietnamworks["VietnamWorks jobs"]
end

subgraph group_data["Persistence and Boards"]
  node_supabase[("Supabase jobs<br/>[schema.sql]")]
  node_vn_store["VN job archive<br/>[vn-jobs.json]"]
  node_topcv_archive["Scraped job archive<br/>[topcv-jobs.json]"]
  node_vn_board["Vietnam job board<br/>[vn.html]"]
  node_us_board["US jobs board<br/>[index.html]"]
end

node_operator(("Job seeker"))
node_schedule["Daily Actions<br/>[crawl-jobs.yml]"]

node_schedule -.->|"runs daily"| node_us_crawler
node_us_crawler -->|"fetches listings"| node_us_feeds
node_us_feeds -->|"returns jobs"| node_us_crawler
node_us_crawler -->|"tags and filters"| node_us_filter
node_us_crawler -->|"upserts jobs"| node_supabase
node_us_crawler -.->|"publishes results"| node_us_board
node_operator -.->|"views listings"| node_vn_board
node_vn_crawler -->|"fetches jobs"| node_vn_linkedin
node_vn_crawler -->|"fetches jobs"| node_itviec
node_vn_crawler -->|"fetches jobs"| node_remote_feeds
node_vn_crawler -->|"classifies listings"| node_vn_enrich
node_vn_crawler -->|"merges and writes"| node_vn_store
node_browser_scraper -->|"scrapes listings"| node_topcv
node_browser_scraper -->|"scrapes listings"| node_vietnamworks
node_browser_scraper -->|"merges and writes"| node_topcv_archive
node_vn_store -.->|"supplies listings"| node_vn_board
node_topcv_archive -.->|"supplies listings"| node_vn_board

click node_schedule "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/.github/workflows/crawl-jobs.yml"
click node_us_crawler "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py"
click node_us_feeds "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py"
click node_us_filter "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py"
click node_supabase "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/schema.sql"
click node_vn_crawler "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_crawl.py"
click node_vn_enrich "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_crawl.py"
click node_vn_store "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/vn-jobs.json"
click node_browser_scraper "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_scrape.js"
click node_topcv_archive "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/topcv-jobs.json"
click node_vn_board "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/vn.html"
click node_us_board "https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_us_crawler,node_us_feeds,node_us_filter toneBlue
class node_vn_crawler,node_vn_linkedin,node_itviec,node_remote_feeds,node_vn_enrich,node_browser_scraper,node_topcv,node_vietnamworks toneAmber
class node_supabase,node_vn_store,node_topcv_archive,node_vn_board,node_us_board toneMint
class node_operator toneIndigo
class node_schedule toneTeal
```

## Components

### US AI Jobs

- [US crawler](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py): `crawl.py`
- [US job feeds](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py): `crawl.py`
- [Role tagging](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/crawl.py): `crawl.py`

### Vietnam .NET Jobs

- [VN crawler](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_crawl.py): `vn_crawl.py`
- **LinkedIn jobs**
- **ITviec jobs**
- **Remote feeds**
- [Fit enrichment](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_crawl.py): `vn_crawl.py`
- [Browser scraper](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/vn_scrape.js): `vn_scrape.js`
- **TopCV jobs**
- **VietnamWorks jobs**

### Persistence and Boards

- [Supabase jobs](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/schema.sql): `schema.sql`
- [VN job archive](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/vn-jobs.json): `docs/vn-jobs.json`
- [Scraped job archive](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/topcv-jobs.json): `docs/topcv-jobs.json`
- [Vietnam job board](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/vn.html): `docs/vn.html`
- [US jobs board](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/docs/index.html): `docs/index.html`

### Other components

- **Job seeker**
- [Daily Actions](https://github.com/tranletrungkien1705/us-ai-jobs-crawler/blob/main/.github/workflows/crawl-jobs.yml): `.github/workflows/crawl-jobs.yml`

## Connections

- Daily Actions → US crawler: runs daily (evidence: `README.md`)
- US crawler → US job feeds: fetches listings (evidence: `crawl.py`)
- US job feeds → US crawler: returns jobs (evidence: `crawl.py`)
- US crawler → Role tagging: tags and filters (evidence: `crawl.py`)
- US crawler → Supabase jobs: upserts jobs (evidence: `crawl.py`)
- US crawler → US jobs board: publishes results (evidence: `README.md`)
- Job seeker → Vietnam job board: views listings
- VN crawler → LinkedIn jobs: fetches jobs (evidence: `vn_crawl.py`)
- VN crawler → ITviec jobs: fetches jobs (evidence: `vn_crawl.py`)
- VN crawler → Remote feeds: fetches jobs (evidence: `vn_crawl.py`)
- VN crawler → Fit enrichment: classifies listings (evidence: `vn_crawl.py`)
- VN crawler → VN job archive: merges and writes (evidence: `vn_crawl.py`)
- Browser scraper → TopCV jobs: scrapes listings (evidence: `vn_scrape.js`)
- Browser scraper → VietnamWorks jobs: scrapes listings (evidence: `vn_scrape.js`)
- Browser scraper → Scraped job archive: merges and writes (evidence: `vn_scrape.js`)
- VN job archive → Vietnam job board: supplies listings
- Scraped job archive → Vietnam job board: supplies listings
