# jhcoco/bosszp-selenium architecture

> Architecture diagram of the GitHub repository jhcoco/bosszp-selenium, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jhcoco/bosszp-selenium
- Repository: https://github.com/jhcoco/bosszp-selenium

## Overview

##### Main Components

a) Data Collection Layer:
- Selenium WebDriver Component
- Boss Website Interface
- Chrome/Browser Component

b) Data Processing Layer:
- Python Scraping Script (boss_selenium.py)
- Data Parsing Component

c) Data Storage Layer:
- MySQL Database (spider_db)
- Database Utility Component (dbutils.py)

**[Boss Website] (External System)**
- ↓
**[Selenium WebDriver]**
- ↓
**[Python Scraping Script]**
- ↓
**[Data Processing]**
- ↓
**[Database Utils]**
- ↓
**[MySQL Database]**

##### Important Elements to Highlight

a) Data Flow:
- How job data moves from the website through the system
- Transformation of raw HTML to structured data
- Database write operations

b) Key Components:
- Selenium WebDriver interaction with the website
- Data parsing and transformation logic
- Database operations through dbutils.py

##### Technical Details

- category
- sub_category
- job_title
- job_location
- job_company
- (other relevant fields)

##### Data transformation steps

Keep the abstraction level focused on the main components while including enough detail to understand the data flow and processing steps. This will help developers understand both the high-level architecture and the specific implementation details of the scraping system.

Remember to indicate that this is a batch processing system rather than a real-time system, as it's designed to collect job posting data in scheduled runs rather than continuous streaming.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph External
        BW[Boss Website]:::external
    end

    subgraph Collection
        SW[Selenium WebDriver]:::driver
        BR[Chrome Browser]:::browser
    end

    subgraph Processing
        PS[Python Scraping Script]:::script
        DP[Data Parser]:::processor
        EH[Error Handler]:::processor
        RL[Rate Limiter]:::processor
    end

    subgraph Storage
        DU[Database Utils]:::database
        DB[(MySQL Database)]:::database
    end

    subgraph Schema
        TS["Table Schema
        - category
        - sub_category
        - job_title
        - job_location
        - job_company"]:::schema
    end

    BW --> SW
    SW --> BR
    BR --> PS
    PS --> DP
    PS --> EH
    PS --> RL
    DP --> DU
    DU --> DB
    DB -.-> TS

    click PS "https://github.com/jhcoco/bosszp-selenium/blob/master/boss_selenium.py"
    click DU "https://github.com/jhcoco/bosszp-selenium/blob/master/dbutils.py"

    classDef external fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef driver fill:#90EE90,stroke:#333,stroke-width:2px
    classDef browser fill:#98FB98,stroke:#333,stroke-width:2px
    classDef script fill:#32CD32,stroke:#333,stroke-width:2px
    classDef processor fill:#3CB371,stroke:#333,stroke-width:2px
    classDef database fill:#FFD700,stroke:#333,stroke-width:2px
    classDef schema fill:#DEB887,stroke:#333,stroke-width:2px

    subgraph Legend
        L1[External System]:::external
        L2[Processing Component]:::processor
        L3[Database Component]:::database
        L4[Schema Information]:::schema
    end
```
