# bunsly/jobspy architecture

> Architecture diagram of the GitHub repository bunsly/jobspy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bunsly/jobspy
- Repository: https://github.com/bunsly/jobspy

## Overview

##### Main Components Identification

The system can be broken down into these primary components:
- Core Scraping Engine
- Individual Job Board Scrapers
- Data Processing Layer
- Public API Interface
- Output Formatting Layer

**Top Level (Client Interface)**
**↓**
**Middle Level (Core Engine & Scrapers)**
**↓**
**Bottom Level (Data Processing & Output)**

##### Specific Component Details

- LinkedIn Scraper
- Indeed Scraper
- Glassdoor Scraper
- Google Jobs Scraper
- ZipRecruiter Scraper

b) Core components:
- Main Scraping Controller (src/jobspy/__init__.py)
- Utility Functions (src/jobspy/scrapers/utils.py)
- Exception Handling (src/jobspy/scrapers/exceptions.py)

##### Connection Flows

- Data flow from scrapers to the central processing unit
- Error handling paths
- Output formatting flow to DataFrame/CSV

##### Specific Elements

**┌─────────────────────────────────────────┐**
- Client Interface             │
- (scrape_jobs() Public Method)      │
- ┘
- ↓
**┌─────────────────────────────────────────┐**
- Scraping Controller             │
- (Parallel Scraping Orchestration)     │
- ┘
- ↙     ↓     ↓     ↓     ↘
- [Individual Scraper Modules]
- Each with:
- Constants
- Scraping Logic
- Rate Limiting
- Proxy Support
- ↓
**┌─────────────────────────────────────────┐**
- Data Processing Layer           │
- (Normalization & Schema Conversion)    │
- ┘
- ↓
**┌─────────────────────────────────────────┐**
- Output Formatting               │
- (DataFrame/CSV Conversion)          │
- ┘

##### Technical Details to Highlight

- Python 3.10+ requirement
- Proxy support system
- Data schema transformations
- Country-specific handling for different job boards
- Rate limiting and blocking prevention mechanisms

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef core fill:#b3d1ff,stroke:#333,stroke-width:2px
    classDef scraper fill:#90EE90,stroke:#333,stroke-width:2px
    classDef utility fill:#FFE5B4,stroke:#333,stroke-width:2px
    classDef error fill:#FFB6C1,stroke:#333,stroke-width:2px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef output fill:#E6E6FA,stroke:#333,stroke-width:2px

    %% Client Interface Layer
    ClientAPI[Public API Interface]:::core
    click ClientAPI "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/jobs/__init__.py"

    %% Core Engine Layer
    CoreEngine[Core Scraping Engine]:::core
    click CoreEngine "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/__init__.py"
    Utils[Utility Functions]:::utility
    click Utils "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/utils.py"
    ErrorHandler[Error Handling System]:::error
    click ErrorHandler "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/exceptions.py"

    %% Scraper Layer
    subgraph Scrapers[Scraper Modules]
        LinkedIn[LinkedIn Scraper]:::scraper
        Indeed[Indeed Scraper]:::scraper
        Glassdoor[Glassdoor Scraper]:::scraper
        Google[Google Jobs Scraper]:::scraper
        ZipRecruiter[ZipRecruiter Scraper]:::scraper
    end

    %% Constants Layer
    subgraph Constants[Scraper Constants]
        LinkedInConst[LinkedIn Constants]:::utility
        IndeedConst[Indeed Constants]:::utility
        GlassdoorConst[Glassdoor Constants]:::utility
        GoogleConst[Google Constants]:::utility
        ZipConst[ZipRecruiter Constants]:::utility
    end

    %% External Services
    subgraph ExternalServices[Job Boards]
        LinkedInExt[LinkedIn]:::external
        IndeedExt[Indeed]:::external
        GlassdoorExt[Glassdoor]:::external
        GoogleExt[Google Jobs]:::external
        ZipExt[ZipRecruiter]:::external
    end

    %% Data Processing and Output
    DataProcess[Data Processing Layer]:::core
    OutputFormat[Output Formatting]:::output

    %% Click Events for Scrapers
    click LinkedIn "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/linkedin/__init__.py"
    click Indeed "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/indeed/__init__.py"
    click Glassdoor "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/glassdoor/__init__.py"
    click Google "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/google/__init__.py"
    click ZipRecruiter "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/ziprecruiter/__init__.py"

    %% Click Events for Constants
    click LinkedInConst "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/linkedin/constants.py"
    click IndeedConst "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/indeed/constants.py"
    click GlassdoorConst "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/glassdoor/constants.py"
    click GoogleConst "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/google/constants.py"
    click ZipConst "https://github.com/Bunsly/JobSpy/blob/main/src/jobspy/scrapers/ziprecruiter/constants.py"

    %% Connections
    ClientAPI --> CoreEngine
    CoreEngine --> Utils
    CoreEngine --> ErrorHandler
    
    CoreEngine --> LinkedIn & Indeed & Glassdoor & Google & ZipRecruiter
    
    LinkedIn --> LinkedInConst
    Indeed --> IndeedConst
    Glassdoor --> GlassdoorConst
    Google --> GoogleConst
    ZipRecruiter --> ZipConst
    
    LinkedIn --> LinkedInExt
    Indeed --> IndeedExt
    Glassdoor --> GlassdoorExt
    Google --> GoogleExt
    ZipRecruiter --> ZipExt
    
    LinkedIn & Indeed & Glassdoor & Google & ZipRecruiter --> DataProcess
    DataProcess --> OutputFormat

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[Scrapers]:::scraper
        L3[Utilities]:::utility
        L4[Error Handling]:::error
        L5[External Services]:::external
        L6[Output]:::output
    end
```
