How abhishek-gawande/job-scraper works
Project type and purpose
• This is a standalone Python-based web-scraping tool (an open-source job-scraper).
• Its goal is to collect listings from multiple job boards (Glassdoor, Indeed, LinkedIn, ZipRecruiter) and dump them into CSV files under datasets_created/.
File structure
• Entry points
– Main.py (possibly a legacy launcher)
– main_scraper.py (the recommended script to run)
• Package layout (jobscrape/)
– jobscrape/__init__.py
– jobscrape/jobs/ ← data models or job-record definitions
– jobscrape/scrapers/ ← scraper strategy implementations
· exceptions.py ← custom error types
· utils.py ← shared helper functions (e.g., HTTP calls, parsers)
· glassdoor/, indeed/, linkedin/, ziprecruiter/ ← site-specific scraper modules
• Output directory
– datasets_created/ ← CSV files generated by runs
• Configuration/build
– requirements.txt ← dependency manifest for pip
– virtual environment invoked per README
README for additional insights
• Implies a simple “orchestrator + strategy” pattern: main_scraper imports each site scraper, invokes its .scrape() method, aggregates results.
Elements
• Clear component labels (e.g., “Orchestrator”, “Glassdoor Scraper”)
– Orchestrator (rectangle)
– Scraper Strategies (ovals)