How kathisnehith/linkedin-jobs-posts-scraper works
Project type and purpose
• This is a Python‐based, open-source web-scraping tool specialized for extracting job listings (“jobs scraper”) and hiring posts (“hirepost scraper”) from LinkedIn. It is not a full‐stack web application but a utilities collection focused on automation, data extraction, and local storage of results.
File structure
• Top-level scripts and notebooks
• Linkedin_hirepost_scrape.ipynb – notebook for exploratory scraping of hiring posts
• Linkedin_jobs_scrape.ipynb – notebook for exploratory scraping of job listings
• README.md – project documentation
• requirements.txt – dependencies list (selenium, beautifulsoup4, pandas, requests, webdriver-manager)
• data/ – sample data folder and prebuilt Chroma email DB zip
• dev-test/ – minimal test harness
• app/ui_jobsearch_test.py – stub UI test script
• backend/linkedinsearch.py – backend test module for scraping logic
• .devcontainer/ – development container config (VS Code)
• Hidden files (.DS_Store) – ignore
README for additional insights
• Two core scripts (in raw .py form, mirrored by notebooks) each:
• Login to LinkedIn with credentials
• Search by keywords, location, posting date
• Parse HTML with BeautifulSoup
• Structure data into pandas DataFrame and export CSV
• Hard-coded parameters (to be parameterized in the future)
• Future work calls out orchestration (Airflow), secret management, keyword filtering, and Google Sheets integration.
• Blue for internal modules (Engine, Parsing, Storage)
• Green for external dependencies (LinkedIn, ChromeDriver)
• Orange for configuration and secrets
• Gray for test harness in dev-test
Additional advice
• Hard-coded parameters as “To Be Configured”
• Reserve space for future modules (Scheduler, Secret Manager, Google Sheets connector) to show planned evolution
• Data directory (data/) is pictured as the local persistence zone, with sample files and zipped DB