# dustadd/scraper architecture

> Architecture diagram of the GitHub repository dustadd/scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dustadd/scraper
- Repository: https://github.com/dustadd/scraper

## Overview

##### Project Type & Purpose
- **Type**: Open-source Python library/tool.
- **Purpose**: Provide a unified API (`scrape_jobs()`) that simultaneously scrapes job postings from multiple job boards (Indeed, LinkedIn, Glassdoor, Google, ZipRecruiter), normalizes the data into a common schema (`JobPost`), and returns a pandas DataFrame.

##### File Structure
- **Top Level**
- `.github/workflows`: CI/CD pipelines for testing and publishing to PyPI
- `pyproject.toml`, `increment_version.py`: Packaging + version bump scripts
- `README.md`, `LICENSE`, pre-commit configs
- **Source Package: `src/jobspy/`**
- `jobs/`: core orchestration or orchestration helpers (likely job models)
- `scrapers/`: one submodule per site (`glassdoor`, `indeed`, `linkedin`, `google`, `ziprecruiter`) + shared `utils.py` + `exceptions.py`
- **Tests:** One test file per scraper + an end-to-end test (`test_all.py`)

##### Key Insights from README
- Central entrypoint: `scrape_jobs(...)`
- Each scraper:
– Fetches HTML/JSON via HTTP (supports proxies)
– Parses site-specific fields into `JobPost` schema
- Returns a pandas DataFrame
- CI pipelines run tests and publish to PyPI

##### Main Components & Interactions
A. **User Layer**
- Python client / script / notebook calls `scrape_jobs`.
B. **API & Orchestrator**
- `scrape_jobs()` function:
– Parses user parameters (sites, search term, filters, proxies)
– Routes calls to individual scraper modules
– Aggregates results
C. **Scraper Submodules (Plugin Architecture)**
- One module per site:
– HTTP client (requests) + optional proxy manager
– Site-specific URL builder & query string logic
– Parsing logic (BeautifulSoup/regex)
- Common utilities:
– `utils.py`: HTML parsing helpers, rate-limit handling
– `exceptions.py`: custom error types
D. **Data Normalization & Output**
- Create `JobPost` objects (title, company, salary, description, etc.)
- Convert list of `JobPost` into pandas DataFrame
E. **External Services**
- Target job board endpoints (Indeed, LinkedIn, Glassdoor, Google Jobs, ZipRecruiter)
- Optional proxy servers for IP rotation
F. **CI/CD & Packaging**
- GitHub Actions:
– `python-test.yml`: run pytest suite
– `publish-to-pypi.yml`: on release, bump version and publish package

##### Architectural Patterns & Design Principles
- **Plugin / Factory Pattern**: Map `site_name` string ➔ corresponding scraper class/module
- **Adapter Pattern**: Each scraper adapts a different HTML structure into a unified `JobPost` schema
- **Pipeline Pattern**: For each site: Fetch ➔ Parse ➔ Normalize ➔ Return
- **Separation of Concerns**:
– API/orchestrator layer vs. individual scraper implementations vs. data‐model layer

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/User Layer
    subgraph "Client/User"
        direction TB
        Client["Client (script/notebook)"]:::client
    end

    %% JobSpy Core Layer
    subgraph "JobSpy Core"
        direction TB
        API["scrape_jobs() EntryPoint"]:::core
        Factory["Scraper Factory"]:::core
        Utils["Shared Utilities"]:::core
        Exceptions["Custom Exceptions"]:::core
        JobPost["JobPost Data Model"]:::core
        Normalizer["Data Normalizer\n(list of JobPost ➔ DataFrame)"]:::core
    end

    %% Scraper Modules Layer
    subgraph "Scraper Modules"
        direction TB
        Glassdoor["Glassdoor Scraper"]:::scraper
        Indeed["Indeed Scraper"]:::scraper
        LinkedIn["LinkedIn Scraper"]:::scraper
        Google["Google Jobs Scraper"]:::scraper
        ZipRecruiter["ZipRecruiter Scraper"]:::scraper
    end

    %% External Services Layer
    subgraph "External Services"
        direction TB
        Boards["Job Board Endpoints\n(Indeed, LinkedIn,\nGlassdoor, Google, ZipRec)"]:::external
        Proxy["Proxy Pool"]:::external
    end

    %% CI/CD & Packaging
    subgraph "CI/CD & Packaging"
        direction TB
        GitTest["GitHub Actions\npython-test.yml"]:::cicd
        GitPublish["GitHub Actions\npublish-to-pypi.yml"]:::cicd
        PyProject["pyproject.toml"]:::cicd
        VersionScript["increment_version.py"]:::cicd
    end

    %% Connections
    Client -->|"calls scrape_jobs()"| API
    API -->|"uses Factory"| Factory
    API -->|"imports JobPost Model"| JobPost
    Factory -->|"loads scraper modules"| Glassdoor
    Factory --> Indeed
    Factory --> LinkedIn
    Factory --> Google
    Factory --> ZipRecruiter

    Glassdoor -->|"HTTP GET\n(parse HTML)"| Boards
    Indeed --> Boards
    LinkedIn --> Boards
    Google --> Boards
    ZipRecruiter --> Boards

    Boards -->|"HTML/JSON response"| Glassdoor
    Boards --> Indeed
    Boards --> LinkedIn
    Boards --> Google
    Boards --> ZipRecruiter

    Glassdoor -->|"parse ➔ JobPost"| Utils
    Indeed --> Utils
    LinkedIn --> Utils
    Google --> Utils
    ZipRecruiter --> Utils

    Glassdoor -->|"errors"| Exceptions
    Indeed --> Exceptions
    LinkedIn --> Exceptions
    Google --> Exceptions
    ZipRecruiter --> Exceptions

    Utils -->|"normalized JobPost"| Normalizer
    JobPost --> Normalizer
    Normalizer -->|"returns DataFrame"| Client

    %% CI/CD flows
    GitTest --> PyProject
    GitTest --> VersionScript
    GitPublish --> PyProject
    GitPublish --> VersionScript

    %% Click Events
    click API "https://github.com/dustadd/scraper/blob/main/src/jobspy/jobs/__init__.py"
    click JobPost "https://github.com/dustadd/scraper/tree/main/src/jobspy/jobs/"
    click Factory "https://github.com/dustadd/scraper/blob/main/src/jobspy/scrapers/__init__.py"
    click Utils "https://github.com/dustadd/scraper/blob/main/src/jobspy/scrapers/utils.py"
    click Exceptions "https://github.com/dustadd/scraper/blob/main/src/jobspy/scrapers/exceptions.py"
    click Glassdoor "https://github.com/dustadd/scraper/tree/main/src/jobspy/scrapers/glassdoor/"
    click Indeed "https://github.com/dustadd/scraper/tree/main/src/jobspy/scrapers/indeed/"
    click LinkedIn "https://github.com/dustadd/scraper/tree/main/src/jobspy/scrapers/linkedin/"
    click Google "https://github.com/dustadd/scraper/tree/main/src/jobspy/scrapers/google/"
    click ZipRecruiter "https://github.com/dustadd/scraper/tree/main/src/jobspy/scrapers/ziprecruiter/"

    %% Styles
    classDef client fill:#dddddd,stroke:#333,stroke-width:1px;
    classDef core fill:#c8e6c9,stroke:#333,stroke-width:1px;
    classDef scraper fill:#bbdefb,stroke:#333,stroke-width:1px;
    classDef external fill:#ffecb3,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef cicd fill:#e1bee7,stroke:#333,stroke-width:1px;
```
