How aykseldi/craiglistjobscrape works
Project Type and Purpose
• Type: A Python-based automation script invoked by a system scheduler.
• Purpose: Periodically scrape a Craigslist jobs page, filter postings by keywords, build an HTML summary, and send it via Gmail API.
Main Components
a. Scheduler Layer
– Cron (and optional rtcwake)
– Responsible for triggering the script every hour.
b. Automation Script (craiglist.py)
– HTTP Client & HTML Parser
• Libraries: requests or urllib + BeautifulSoup
– Filter Logic
• Matches job titles against a configured keyword list
– HTML Report Generator
• Assembles filtered results into an email-ready HTML body
– Gmail API Client
• Google API credentials, OAuth2 flow, sendMessage endpoint
c. External Services
– Craigslist Website (HTTP endpoint)
– Google Gmail API (REST endpoint)
d. Recipients
– Target email address(es) receiving the report
Relationships and Data Flow
• Cron → (executes) → craiglist.py
• craiglist.py → (HTTP GET) → Craigslist
• Craigslist → (HTML) → craiglist.py
• craiglist.py → (parse/filter/generate) → HTML body
• craiglist.py → (HTTPS POST) → Gmail API
• Gmail API → (delivers email) → Recipient inbox
• (Optional) craiglist.py → (invokes) → rtcwake → (puts system to sleep)
Architectural Patterns and Design Principles
– Pipeline pattern: sequence of fetch → parse → filter → generate → send
– Loose coupling: the scheduler, scraping logic, and email dispatch are separate steps
– Reusability: the script could be extended to support multiple keyword sets or output formats
Technologies / Frameworks / Libraries
– Python 3 (core language)
– requests or urllib + BeautifulSoup (web scraping)
– google-api-python-client & oauth2client (Gmail integration)
– cron & rtcwake (scheduling & power management)