# wajih1234/job-scraper architecture

> Architecture diagram of the GitHub repository wajih1234/job-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wajih1234/job-scraper
- Repository: https://github.com/wajih1234/job-scraper

## Overview

This is a small Tunisian job-listing scraper with a Streamlit dashboard. The main user workflow searches Tanitjobs for a job title, collects and parses matching offers, saves them to SQLite, then queries the database for title matches and filters by city. A separate batch workflow crawls links, scrapes offers into JSON, and imports them into the same database; a command-line script searches that database by title or description. The README documents the dashboard and local database; the batch workflow is evident in the sampled scripts.

Main entry points: [`app.py`](https://github.com/wajih1234/job-scraper/blob/main/app.py) (Streamlit Dashboard), [`scraper.py`](https://github.com/wajih1234/job-scraper/blob/main/scraper.py) (Batch Parser), [`tanit.py`](https://github.com/wajih1234/job-scraper/blob/main/tanit.py) (Offer Writer), [`get_links.py`](https://github.com/wajih1234/job-scraper/blob/main/get_links.py) (Page Crawler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_discovery["Job Discovery"]
  node_keyword_search["Keyword Search<br/>[tanit.py]"]
  node_page_crawler["Page Crawler<br/>[get_links.py]"]
end

subgraph group_ingestion["Offer Ingestion"]
  node_offer_parser["Offer Parser<br/>[tanit.py]"]
  node_batch_parser["Batch Parser<br/>[scraper.py]"]
end

subgraph group_storage["Local Storage"]
  node_offer_writer["Offer Writer<br/>[tanit.py]"]
  node_json_importer["JSON Importer<br/>[to_db.py]"]
  node_jobs_db[("Jobs Database")]
end

subgraph group_interfaces["Search Interfaces"]
  node_dashboard["Streamlit Dashboard<br/>[app.py]"]
  node_cli_search["CLI Search<br/>[search.py]"]
end

node_jobseeker(("Job Seeker"))
node_tanit_site(("Tanitjobs"))

node_jobseeker -->|"uses"| node_dashboard
node_dashboard -->|"requests offers"| node_keyword_search
node_keyword_search -->|"fetches listings"| node_tanit_site
node_dashboard -->|"saves links"| node_offer_writer
node_offer_writer -->|"parses offers"| node_offer_parser
node_offer_parser -->|"fetches details"| node_tanit_site
node_offer_writer -->|"writes offers"| node_jobs_db
node_dashboard -->|"queries and filters"| node_jobs_db
node_jobseeker -->|"searches offers"| node_cli_search
node_cli_search -->|"queries offers"| node_jobs_db
node_page_crawler -->|"crawls listings"| node_tanit_site
node_page_crawler -->|"writes links for"| node_batch_parser
node_batch_parser -->|"fetches details"| node_tanit_site
node_batch_parser -->|"writes offers for"| node_json_importer
node_json_importer -->|"imports offers"| node_jobs_db

click node_dashboard "https://github.com/wajih1234/job-scraper/blob/main/app.py"
click node_cli_search "https://github.com/wajih1234/job-scraper/blob/main/search.py"
click node_keyword_search "https://github.com/wajih1234/job-scraper/blob/main/tanit.py"
click node_page_crawler "https://github.com/wajih1234/job-scraper/blob/main/get_links.py"
click node_offer_parser "https://github.com/wajih1234/job-scraper/blob/main/tanit.py"
click node_batch_parser "https://github.com/wajih1234/job-scraper/blob/main/scraper.py"
click node_offer_writer "https://github.com/wajih1234/job-scraper/blob/main/tanit.py"
click node_json_importer "https://github.com/wajih1234/job-scraper/blob/main/to_db.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_keyword_search,node_page_crawler toneBlue
class node_offer_parser,node_batch_parser toneAmber
class node_offer_writer,node_json_importer,node_jobs_db toneMint
class node_dashboard,node_cli_search toneRose
class node_jobseeker,node_tanit_site toneIndigo
```

## Components

### Job Discovery

- [Keyword Search](https://github.com/wajih1234/job-scraper/blob/main/tanit.py): `tanit.py`
- [Page Crawler](https://github.com/wajih1234/job-scraper/blob/main/get_links.py): `get_links.py`

### Offer Ingestion

- [Offer Parser](https://github.com/wajih1234/job-scraper/blob/main/tanit.py): `tanit.py`
- [Batch Parser](https://github.com/wajih1234/job-scraper/blob/main/scraper.py): `scraper.py`

### Local Storage

- [Offer Writer](https://github.com/wajih1234/job-scraper/blob/main/tanit.py): `tanit.py`
- [JSON Importer](https://github.com/wajih1234/job-scraper/blob/main/to_db.py): `to_db.py`
- **Jobs Database**

### Search Interfaces

- [Streamlit Dashboard](https://github.com/wajih1234/job-scraper/blob/main/app.py): `app.py`
- [CLI Search](https://github.com/wajih1234/job-scraper/blob/main/search.py): `search.py`

### Other components

- **Job Seeker**
- **Tanitjobs**

## Connections

- Job Seeker → Streamlit Dashboard: uses (evidence: `README.md`)
- Streamlit Dashboard → Keyword Search: requests offers (evidence: `app.py`)
- Keyword Search → Tanitjobs: fetches listings (evidence: `tanit.py`)
- Streamlit Dashboard → Offer Writer: saves links (evidence: `app.py`)
- Offer Writer → Offer Parser: parses offers (evidence: `tanit.py`)
- Offer Parser → Tanitjobs: fetches details (evidence: `tanit.py`)
- Offer Writer → Jobs Database: writes offers (evidence: `tanit.py`)
- Streamlit Dashboard → Jobs Database: queries and filters (evidence: `app.py`)
- Job Seeker → CLI Search: searches offers (evidence: `search.py`)
- CLI Search → Jobs Database: queries offers (evidence: `search.py`)
- Page Crawler → Tanitjobs: crawls listings (evidence: `get_links.py`)
- Page Crawler → Batch Parser: writes links for (evidence: `get_links.py`)
- Batch Parser → Tanitjobs: fetches details (evidence: `scraper.py`)
- Batch Parser → JSON Importer: writes offers for (evidence: `scraper.py`)
- JSON Importer → Jobs Database: imports offers (evidence: `to_db.py`)
