How dustadd/scraper works
Project Type & Purpose
• Type: Open-source Python library/tool.
• Purpose: Provide a unified API (scrape_jobs()) that simultaneously scrapes job postings from multiple job boards (Indeed, LinkedIn, Glassdoor, Google, ZipRecruiter), normalizes the data into a common schema (JobPost), and returns a pandas DataFrame.
File Structure
• Top Level
• .github/workflows: CI/CD pipelines for testing and publishing to PyPI
• pyproject.toml, increment_version.py: Packaging + version bump scripts
• README.md, LICENSE, pre-commit configs
• Source Package: src/jobspy/
• jobs/: core orchestration or orchestration helpers (likely job models)
• scrapers/: one submodule per site (glassdoor, indeed, linkedin, google, ziprecruiter) + shared utils.py + exceptions.py
• Tests: One test file per scraper + an end-to-end test (test_all.py)
Key Insights from README
• Central entrypoint: scrape_jobs(...)
• Each scraper:
– Fetches HTML/JSON via HTTP (supports proxies)
– Parses site-specific fields into JobPost schema
• Returns a pandas DataFrame
• CI pipelines run tests and publish to PyPI
Main Components & Interactions
A. User Layer
• Python client / script / notebook calls scrape_jobs.
B. API & Orchestrator
• scrape_jobs() function:
– Parses user parameters (sites, search term, filters, proxies)
– Routes calls to individual scraper modules
– Aggregates results
C. Scraper Submodules (Plugin Architecture)
• One module per site:
– HTTP client (requests) + optional proxy manager
– Site-specific URL builder & query string logic
– Parsing logic (BeautifulSoup/regex)
• Common utilities:
– utils.py: HTML parsing helpers, rate-limit handling
– exceptions.py: custom error types
D. Data Normalization & Output
• Create JobPost objects (title, company, salary, description, etc.)
• Convert list of JobPost into pandas DataFrame
E. External Services
• Target job board endpoints (Indeed, LinkedIn, Glassdoor, Google Jobs, ZipRecruiter)
• Optional proxy servers for IP rotation
F. CI/CD & Packaging
• GitHub Actions:
– python-test.yml: run pytest suite
– publish-to-pypi.yml: on release, bump version and publish package
Architectural Patterns & Design Principles
• Plugin / Factory Pattern: Map site_name string ➔ corresponding scraper class/module
• Adapter Pattern: Each scraper adapts a different HTML structure into a unified JobPost schema
• Pipeline Pattern: For each site: Fetch ➔ Parse ➔ Normalize ➔ Return
• Separation of Concerns:
– API/orchestrator layer vs. individual scraper implementations vs. data‐model layer