# kathisnehith/linkedin-jobs-posts-scraper architecture

> Architecture diagram of the GitHub repository kathisnehith/linkedin-jobs-posts-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kathisnehith/linkedin-jobs-posts-scraper
- Repository: https://github.com/kathisnehith/linkedin-jobs-posts-scraper

## Overview

##### Project type and purpose
- This is a Python‐based, open-source web-scraping tool specialized for extracting job listings (“jobs scraper”) and hiring posts (“hirepost scraper”) from LinkedIn. It is not a full‐stack web application but a utilities collection focused on automation, data extraction, and local storage of results.

##### File structure
- Top-level scripts and notebooks
- Linkedin_hirepost_scrape.ipynb – notebook for exploratory scraping of hiring posts
- Linkedin_jobs_scrape.ipynb – notebook for exploratory scraping of job listings
- README.md – project documentation
- requirements.txt – dependencies list (selenium, beautifulsoup4, pandas, requests, webdriver-manager)
- data/ – sample data folder and prebuilt Chroma email DB zip
- dev-test/ – minimal test harness
- app/ui_jobsearch_test.py – stub UI test script
- backend/linkedinsearch.py – backend test module for scraping logic
- .devcontainer/ – development container config (VS Code)
- Hidden files (.DS_Store) – ignore

##### README for additional insights
- Two core scripts (in raw .py form, mirrored by notebooks) each:
- Login to LinkedIn with credentials
- Search by keywords, location, posting date
- Parse HTML with BeautifulSoup
- Structure data into pandas DataFrame and export CSV
- Hard-coded parameters (to be parameterized in the future)
- Future work calls out orchestration (Airflow), secret management, keyword filtering, and Google Sheets integration.

- Blue for internal modules (Engine, Parsing, Storage)
- Green for external dependencies (LinkedIn, ChromeDriver)
- Orange for configuration and secrets
- Gray for test harness in dev-test

##### Additional advice
- Hard-coded parameters as “To Be Configured”
- Reserve space for future modules (Scheduler, Secret Manager, Google Sheets connector) to show planned evolution
- Data directory (data/) is pictured as the local persistence zone, with sample files and zipped DB

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "User Interface / Invocation Layer"
        UI1["Linkedin_hirepost_scrape.ipynb"]:::internal
        UI2["Linkedin_jobs_scrape.ipynb"]:::internal
    end

    subgraph "Scraper Engine"
        E1["HirePostScraper Module"]:::internal
        E2["JobListingScraper Module"]:::internal
    end

    subgraph "Browser Automation Layer"
        Auto["Selenium WebDriver<br/>ChromeDriver"]:::external
    end

    subgraph "HTML Parsing Layer"
        Parse["BeautifulSoup Parser"]:::internal
    end

    subgraph "Data Processing & Storage"
        DF["pandas DataFrame"]:::internal
        CSV["CSV Sink"]:::internal
        DB["Chroma Email DB (Zip)"]:::external
        Sample["Sample Data Folder"]:::external
    end

    subgraph "Test Harness"
        T1["ui_jobsearch_test.py"]:::test
        T2["linkedinsearch.py (Test)"]:::test
    end

    subgraph "Configuration & Development Environment"
        Config1["requirements.txt"]:::config
        Config2[".devcontainer/devcontainer.json"]:::config
        Doc["README.md"]:::config
    end

    LinkedIn["LinkedIn Website"]:::external

    UI1 -->|invokes| E1
    UI2 -->|invokes| E2
    E1 -->|drives| Auto
    E2 -->|drives| Auto
    Auto -->|requests/DOM| LinkedIn
    LinkedIn -->|returns HTML| Auto
    E1 -->|raw HTML| Parse
    E2 -->|raw HTML| Parse
    Parse -->|structured data| DF
    DF -->|export| CSV
    DF -->|store| DB
    DF -->|store| Sample
    T1 -.->|tests UI flows| E2
    T2 -.->|tests engine| E1
    T2 -.->|tests engine| E2
    Config1 -.-> Auto
    Config1 -.-> Parse
    Config1 -.-> E1
    Config1 -.-> E2

    click UI1 "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/Linkedin_hirepost_scrape.ipynb"
    click UI2 "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/Linkedin_jobs_scrape.ipynb"
    click T1 "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/dev-test/app/ui_jobsearch_test.py"
    click T2 "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/dev-test/backend/linkedinsearch.py"
    click Config1 "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/requirements.txt"
    click Config2 "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/.devcontainer/devcontainer.json"
    click Doc "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/README.md"
    click DB "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/blob/main/data/chroma_email_db.zip"
    click Sample "https://github.com/kathisnehith/linkedin-jobs-posts-scraper/tree/main/data/sample"

    classDef internal fill:#ADD8E6,stroke:#000;
    classDef external fill:#90EE90,stroke:#000;
    classDef config fill:#FFA500,stroke:#000;
    classDef test fill:#D3D3D3,stroke:#000;
```
