# cwwmbm/linkedinscraper architecture

> Architecture diagram of the GitHub repository cwwmbm/linkedinscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cwwmbm/linkedinscraper
- Repository: https://github.com/cwwmbm/linkedinscraper

## Overview

##### Main Components
- Scraper Module (main.py): This component reads the configuration from config.json (with config_example.json as a template), performs web scraping using libraries like Requests and BeautifulSoup, processes job postings (applying include/exclude filters), and stores unique records in the SQLite database.
- Web Interface Module (app.py): The Flask-powered web application that serves HTML pages (using the templates directory such as job_description.html and jobs.html) and static assets (found in the static directory, e.g., job_actions.js) to allow users to view and modify job statuses.
- Database: An SQLite database that holds job postings. The configuration parameters include table names (jobs_tablename, filtered_jobs_tablename) and the db_path. This is the persistent storage layer used by both the scraper and the web interface.
- External API Integration: The new OpenAI integration for cover letter generation. This is configured via the config.json parameters (OpenAI_API_KEY, OpenAI_Model, and resume_path) and is used to enhance the scrapper’s output or provide additional functionality with generated cover letters.
- Configuration: The config.json file (and example file) which contains all the runtime parameters for proxies, user-agents, search queries, keyword filters, etc.
- Testing Infrastructure: The tests directory (e.g., tests/test_proxy_connection.py) that ensures critical functionalities like proxy connectivity are working as expected.

##### Relationships and Interactions
- The Scraper (main.py) initiates HTTP requests—potentially through proxy servers, as indicated in the config—and processes the HTML using BeautifulSoup. It then applies filtering criteria (keywords, languages, timespan, etc.) before inserting job postings into the SQLite database.
- The Flask web application (app.py) reads data from the same SQLite database to display job postings on the web interface. It also accepts user interactions (marking jobs as applied, rejected, etc.) and writes the status back to the database.
- The OpenAI integration overlays the system by taking a resume (PDF), sending it to the OpenAI API for cover letter generation, and then integrating the result into the application workflow.
- The configuration file (config.json) acts as a central control point. Both the scraper and web interface read from it to adjust behaviors (like proxy settings, search queries, and API keys).

##### Architectural Patterns and Design Principles
- Separation of Concerns: The project clearly separates job data scraping from user interface functions. This is evident with distinct entry scripts (main.py for scraping versus app.py for serving the web UI).
- Modularization: The project structure—having separate directories for tests, static files, and templates—supports clean separation between presentation, logic, and configuration.
- Client-Server Model: Flask serves as the web server (backend) communicating over HTTP with a client browser, emphasizing stateless web interactions.
- External API Integration: The use of OpenAI features a service integration pattern, where the system interacts with an external service for advanced processing (cover letter generation).

##### Additional Details
- To capture the nuances of the job filtering logic by noting that the scraper removes duplicates and irrelevant postings based on titles, descriptions, and even languages.
- Notes that the system is designed to be run in two phases (first running main.py to populate the database, and then app.py for the UI), so a sequential flow between these phases should be indicated.
- Optionally, include the testing module as a parallel component (show dependency on proxy testing), which confirms system reliability.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Testing Layer
    Config["Configuration (config.json)"]:::config
    ProxyTest["Proxy Testing"]:::test

    %% Backend Modules
    Scraper["Scraper Module (main.py)"]:::core
    WebApp["Web Interface Module (app.py)"]:::core

    %% Data Storage
    Database["SQLite Database"]:::db

    %% External Systems
    LinkedIn["LinkedIn"]:::external
    OpenAI["OpenAI API"]:::external

    %% Frontend Layer
    Browser["Client Browser"]:::ui
    TemplateDesc["Job Description Template (HTML)"]:::ui
    TemplateJobs["Jobs Template (HTML)"]:::ui
    StaticAsset["Static Asset (job_actions.js)"]:::ui

    %% Relationships
    Config -->|"feeds"| Scraper
    Config -->|"feeds"| WebApp

    Scraper -->|"ScrapeJobPostings"| LinkedIn
    Scraper -->|"StoreJobData"| Database
    Scraper -->|"CoverLetterGeneration"| OpenAI

    WebApp -->|"Fetch/UpdateJobData"| Database
    WebApp -->|"ServeUI"| Browser
    WebApp -->|"RenderHTML"| TemplateDesc
    WebApp -->|"RenderHTML"| TemplateJobs
    WebApp -->|"LoadAssets"| StaticAsset

    ProxyTest -->|"ValidatesProxy"| Scraper

    %% Click Events
    click Scraper "https://github.com/cwwmbm/linkedinscraper/blob/main/main.py"
    click WebApp "https://github.com/cwwmbm/linkedinscraper/blob/main/app.py"
    click Config "https://github.com/cwwmbm/linkedinscraper/blob/main/config_example.json"
    click TemplateDesc "https://github.com/cwwmbm/linkedinscraper/blob/main/templates/job_description.html"
    click TemplateJobs "https://github.com/cwwmbm/linkedinscraper/blob/main/templates/jobs.html"
    click StaticAsset "https://github.com/cwwmbm/linkedinscraper/blob/main/static/job_actions.js"
    click ProxyTest "https://github.com/cwwmbm/linkedinscraper/blob/main/tests/test_proxy_connection.py"

    %% Styles
    classDef config fill:#ffefc6,stroke:#bf5700,stroke-width:2px
    classDef core fill:#cce5ff,stroke:#004085,stroke-width:2px
    classDef db fill:#fff3cd,stroke:#856404,stroke-width:2px
    classDef external fill:#d4edda,stroke:#155724,stroke-width:2px
    classDef ui fill:#f8d7da,stroke:#721c24,stroke-width:2px
    classDef test fill:#f5c6cb,stroke:#721c24,stroke-width:2px
```
