How cqawam/jober works
Project type and purpose
• This is a standalone Python automation tool (“Jober”) whose sole purpose is to scrape entry‑level cybersecurity job listings from the iSecJobs website at regular intervals and persist them to timestamped text files.
File structure
• Top‑level files:
• README.md (documentation)
• jober.py (main script)
• job_posts/ (output directory containing previous job dumps)
• No subdirectories beyond the output folder; this indicates a single‑process script with no separate frontend/backend.
README for additional insights
• Dependencies: requests, BeautifulSoup4
• Key variables: waiting_time (interval), output_dir (target folder)
• Future scope: multiple sites, structured storage, notification system
Main components
a. Components
• Scheduler/Loop (internal timer or external cron job)
• Jober Script
• HTTP Fetcher Module (uses requests)
• HTML Parser Module (uses BeautifulSoup4)
• Data Transformer (extracts title, company, location, skills, salary, link)
• Storage Module (writes to timestamped files in job_posts/)
• Error Handler/Logger
• External Service
• iSecJobs Website (target of HTTP GET requests)
• File System
• Local directory job_posts/ for output text files
b. Relationships and Data Flow
1. Scheduler triggers Jober Script
2. Script → HTTP Fetcher → GET https://isecjobs.com → receives HTML
3. HTML → Parser → raw job listings
4. Parsed data → Transformer → structured records
5. Records → Storage Module → write to timestamped .txt in job_posts/
6. Errors → Error Handler → logs or retries
c. Architectural Patterns / Design Principles
• Separation of concerns: distinct modules handle fetching, parsing, transformation, storage
• Extendability hooks: easy to swap out or add parsers/targets
• Single process, sequential workflow
d. Key Technologies
• Python 3.x
• requests library (HTTP client)
• BeautifulSoup4 (HTML parsing)
• Native file I/O for storage