# abhishek-gawande/job-scraper architecture

> Architecture diagram of the GitHub repository abhishek-gawande/job-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhishek-gawande/job-scraper
- Repository: https://github.com/abhishek-gawande/job-scraper

## Overview

##### Project type and purpose
- This is a standalone Python-based web-scraping tool (an open-source job-scraper).
- Its goal is to collect listings from multiple job boards (Glassdoor, Indeed, LinkedIn, ZipRecruiter) and dump them into CSV files under datasets_created/.

##### File structure
- Entry points
– Main.py (possibly a legacy launcher)
– main_scraper.py (the recommended script to run)
- Package layout (jobscrape/)
– jobscrape/__init__.py
– jobscrape/jobs/             ← data models or job-record definitions
– jobscrape/scrapers/         ← scraper strategy implementations
· exceptions.py             ← custom error types
· utils.py                  ← shared helper functions (e.g., HTTP calls, parsers)
· glassdoor/, indeed/, linkedin/, ziprecruiter/ ← site-specific scraper modules
- Output directory
– datasets_created/          ← CSV files generated by runs
- Configuration/build
– requirements.txt           ← dependency manifest for pip
– virtual environment invoked per README

##### README for additional insights
- Implies a simple “orchestrator + strategy” pattern: main_scraper imports each site scraper, invokes its .scrape() method, aggregates results.

##### Elements
- Clear component labels (e.g., “Orchestrator”, “Glassdoor Scraper”)
– Orchestrator (rectangle)
– Scraper Strategies (ovals)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Application Entry Points"
        Legacy["Legacy Launcher\nMain.py"]:::config
        Orchestrator["Orchestrator\nmain_scraper.py"]:::orchestrator
    end

    %% Core Package
    subgraph "jobscrape Package"
        InitPkg["Package Init\n__init__.py"]:::config
        subgraph "Scrapers Module"
            ScrapersInit["Scrapers Init\n__init__.py"]:::config
            Factory["Scraper Factory\nutils.py"]:::factory
            Exceptions["Shared Exceptions\nexceptions.py"]:::exceptions
            Utils["Shared Utils\nutils.py"]:::utils
            subgraph "Site Scrapers"
                Glassdoor["Glassdoor Scraper"]:::scraper
                Indeed["Indeed Scraper"]:::scraper
                LinkedIn["LinkedIn Scraper"]:::scraper
                ZipRecruiter["ZipRecruiter Scraper"]:::scraper
            end
        end
        Models["Data Models\njobs/"]:::model
    end

    %% External Services
    GlassdoorEXT["Glassdoor API"]:::external
    IndeedEXT["Indeed API"]:::external
    LinkedInEXT["LinkedIn API"]:::external
    ZipEXT["ZipRecruiter API"]:::external

    %% Storage and Config
    Storage["CSV Storage\ndatasets_created/"]:::storage
    Config["Dependencies\nrequirements.txt"]:::config

    %% Relationships
    Legacy --> Orchestrator
    Orchestrator --> Factory
    Factory --> Glassdoor
    Factory --> Indeed
    Factory --> LinkedIn
    Factory --> ZipRecruiter
    Glassdoor --> GlassdoorEXT
    Indeed --> IndeedEXT
    LinkedIn --> LinkedInEXT
    ZipRecruiter --> ZipEXT
    Glassdoor --> Utils
    Indeed --> Utils
    LinkedIn --> Utils
    ZipRecruiter --> Utils
    Glassdoor --> Exceptions
    Indeed --> Exceptions
    LinkedIn --> Exceptions
    ZipRecruiter --> Exceptions
    Glassdoor --> Models
    Indeed --> Models
    LinkedIn --> Models
    ZipRecruiter --> Models
    Orchestrator --> Storage
    Orchestrator --> Config

    %% Click Events
    click Orchestrator "https://github.com/abhishek-gawande/job-scraper/blob/main/main_scraper.py"
    click Legacy "https://github.com/abhishek-gawande/job-scraper/blob/main/Main.py"
    click Factory "https://github.com/abhishek-gawande/job-scraper/blob/main/jobscrape/scrapers/utils.py"
    click Exceptions "https://github.com/abhishek-gawande/job-scraper/blob/main/jobscrape/scrapers/exceptions.py"
    click Models "https://github.com/abhishek-gawande/job-scraper/tree/main/jobscrape/jobs/"
    click Glassdoor "https://github.com/abhishek-gawande/job-scraper/tree/main/jobscrape/scrapers/glassdoor/"
    click Indeed "https://github.com/abhishek-gawande/job-scraper/tree/main/jobscrape/scrapers/indeed/"
    click LinkedIn "https://github.com/abhishek-gawande/job-scraper/tree/main/jobscrape/scrapers/linkedin/"
    click ZipRecruiter "https://github.com/abhishek-gawande/job-scraper/tree/main/jobscrape/scrapers/ziprecruiter/"
    click Storage "https://github.com/abhishek-gawande/job-scraper/tree/main/datasets_created/"
    click Config "https://github.com/abhishek-gawande/job-scraper/blob/main/requirements.txt"
    click InitPkg "https://github.com/abhishek-gawande/job-scraper/blob/main/jobscrape/__init__.py"
    click ScrapersInit "https://github.com/abhishek-gawande/job-scraper/blob/main/jobscrape/scrapers/__init__.py"

    %% Styles
    classDef orchestrator fill:#D6EAF8,stroke:#2E86C1,color:#1B4F72,stroke-width:2px;
    classDef factory fill:#FEF9E7,stroke:#B7950B,color:#7D6608,stroke-width:2px;
    classDef scraper fill:#E8F8F5,stroke:#148F77,color:#0E6251,stroke-width:2px,shape:ellipse;
    classDef utils fill:#FCF3CF,stroke:#B9770E,color:#784212,stroke-width:2px;
    classDef exceptions fill:#FDEDEC,stroke:#C0392B,color:#641E16,stroke-width:2px;
    classDef model fill:#E8DAEF,stroke:#6C3483,color:#4A235A,stroke-width:2px;
    classDef external fill:#F2F3F4,stroke:#7B7D7D,color:#2C3E50,stroke-width:2px,shape:cloud;
    classDef storage fill:#F5ECCE,stroke:#B9770E,color:#784212,stroke-width:2px,shape:cylinder;
    classDef config fill:#EBF5FB,stroke:#1F618D,color:#1A5276,stroke-width:2px;
```
