# munhyok/gamlendar_selenium architecture

> Architecture diagram of the GitHub repository munhyok/gamlendar_selenium, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/munhyok/gamlendar_selenium
- Repository: https://github.com/munhyok/gamlendar_selenium

## Overview

##### Project Type and Purpose
- This is a Python-based ETL scraping tool (“Gamlendar Selenium”) that automatically collects upcoming game release data from multiple platforms (Steam, Playstation, Xbox, Switch), cleans the raw data, backs it up as CSVs, logs failures, and loads the cleaned data into MariaDB (and then to MongoDB). It uses Selenium (via a WebDriver singleton) for data collection.

##### Top-Level Components
- External Services / Data Sources (Steam, Playstation, Xbox, Switch websites)
- Scraper Modules (one per platform under upcoming/)
- Core Pipeline
- WebDriver manager (core/Webdriver.py)
- Data layer (core/data/…): concatData, csvSearch, dataInsert
- Cleaning layer (core/data_cleaning/DataCleaning.py)
- Database layer (core/database/Database.py, MariaDB + MongoDB sync)
- Logging (core/logs/failedLog.py)
- Notifications (core/notification/push_notification.py)
- File Storage (backup CSV folders under upcoming/…/backup and test/result)
- Databases (MariaDB, MongoDB)
- Orchestration / Entrypoint (main.py, playstation_main.py, steam_main.py, etc.)
- Tests (test/)

##### Relationships and Data Flow
1. Orchestrator (main.py or each platform main) invokes the WebDriver singleton.
2. The Scraper Module (pageScrap.py → detailScrap.py) uses Selenium to crawl the platform’s upcoming list pages and detail pages on External Services.
3. Raw data is written to File Storage as timestamped CSV backups.
4. On each fetch or in a batch, Core data layer reads the backups:
a. concatData.py merges multiple CSVs
b. csvSearch.py looks up existing records
c. dataInsert.py writes cleaned records into Database layer
5. core/data_cleaning/DataCleaning.py processes strings, dates, filters out adult titles, normalizes names.
6. core/database/Database.py (singleton) handles writing to MariaDB, then optionally syncing to MongoDB.
7. core/logs/failedLog.py catches and persists scraping/processing errors into per-platform log files.
8. core/notification/push_notification.py can send alerts on critical failures or new releases.

##### Architectural Patterns and Principles
- ETL pipeline pattern: Extract (Selenium scrapers) → Transform (data cleaning) → Load (MariaDB/MongoDB).
- Singleton pattern: WebDriver manager and Database connector use a single instance.
- Separation of concerns: platform-specific scrapers vs. shared core functionality.
- Modular file backup storage for resiliency and replayability.
- Layered architecture: UI (Selenium), Service (core pipeline), Persistence (DB).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "External Services"
        steamSite["Steam Website"]:::external
        playStationSite["PlayStation Website"]:::external
        xboxSite["Xbox Website"]:::external
        switchSite["Switch Website"]:::external
    end

    %% Orchestrator
    orchestrator["main.py"]:::core

    %% Scraper Modules
    subgraph "Scraper Modules"
        subgraph "Steam Scraper"
            steamMain["steam_main.py"]:::scraper
            steamPage["pageScrap.py"]:::scraper
            steamScroll["scrollScrap.py"]:::scraper
            steamDetail["detailScrap.py"]:::scraper
            steamBackup["upcoming/steam/backup/"]:::storage
            steamLog["upcoming/steam/log/"]:::storage
        end
        subgraph "PlayStation Scraper"
            psMain["playstation_main.py"]:::scraper
            psPage["pageScrap.py"]:::scraper
            psDetail["detailScrap.py"]:::scraper
            psBackup["upcoming/playstation/backup/"]:::storage
            psLog["upcoming/playstation/log/"]:::storage
        end
        subgraph "Xbox Scraper"
            xboxMain["xbox_main.py"]:::scraper
            xboxPage["pageScrap.py"]:::scraper
            xboxDetail["detailScrap.py"]:::scraper
            xboxBackup["upcoming/xbox/backup/"]:::storage
            xboxLog["upcoming/xbox/log/"]:::storage
        end
        subgraph "Switch Scraper"
            switchMain["switch_main.py"]:::scraper
            switchPage["pageScrap.py"]:::scraper
            switchDetail["detailScrap.py"]:::scraper
            switchBackup["upcoming/switch/backup/"]:::storage
            switchLog["upcoming/switch/log/"]:::storage
        end
    end

    %% File Storage for tests
    testResult["test/result/"]:::storage

    %% Core Pipeline
    subgraph "Core Pipeline"
        webdriver["WebDriver Singleton"]:::core
        concatData["concatData.py"]:::core
        csvSearch["csvSearch.py"]:::core
        dataCleaning["DataCleaning.py"]:::core
        dataInsert["dataInsert.py"]:::core
        dbConnector["Database Connector"]:::core
        failedLog["failedLog.py"]:::core
        notification["push_notification.py"]:::notification
    end

    %% Databases
    mariadb[(MariaDB)]:::database
    mongodb[(MongoDB)]:::database

    %% Relationships
    orchestrator -->|invokes| webdriver

    steamSite -->|scrape list| steamMain
    playStationSite -->|scrape list| psMain
    xboxSite -->|scrape list| xboxMain
    switchSite -->|scrape list| switchMain

    orchestrator -->|start| steamMain
    orchestrator -->|start| psMain
    orchestrator -->|start| xboxMain
    orchestrator -->|start| switchMain

    steamMain -->|calls| steamPage
    steamPage --> steamScroll
    steamScroll --> steamDetail
    steamDetail -->|raw CSV →| steamBackup
    steamDetail -->|errors →| steamLog

    psMain --> psPage
    psPage --> psDetail
    psDetail -->|raw CSV →| psBackup
    psDetail -->|errors →| psLog

    xboxMain --> xboxPage
    xboxPage --> xboxDetail
    xboxDetail -->|raw CSV →| xboxBackup
    xboxDetail -->|errors →| xboxLog

    switchMain --> switchPage
    switchPage --> switchDetail
    switchDetail -->|raw CSV →| switchBackup
    switchDetail -->|errors →| switchLog

    testResult --> concatData

    steamBackup --> concatData
    psBackup --> concatData
    xboxBackup --> concatData
    switchBackup --> concatData

    concatData --> csvSearch
    csvSearch --> dataCleaning
    dataCleaning --> dataInsert
    dataInsert --> dbConnector
    dbConnector --> mariadb
    dbConnector --> mongodb

    steamLog --> failedLog
    psLog --> failedLog
    xboxLog --> failedLog
    switchLog --> failedLog
    dataInsert -->|errors →| failedLog

    failedLog --> notification
    dbConnector -->|new release →| notification

    %% Click Events
    click webdriver "https://github.com/munhyok/gamlendar_selenium/blob/main/core/Webdriver.py"
    click concatData "https://github.com/munhyok/gamlendar_selenium/blob/main/core/data/concatData.py"
    click csvSearch "https://github.com/munhyok/gamlendar_selenium/blob/main/core/data/csvSearch.py"
    click dataInsert "https://github.com/munhyok/gamlendar_selenium/blob/main/core/data/dataInsert.py"
    click dataCleaning "https://github.com/munhyok/gamlendar_selenium/blob/main/core/data_cleaning/DataCleaning.py"
    click dbConnector "https://github.com/munhyok/gamlendar_selenium/blob/main/core/database/Database.py"
    click failedLog "https://github.com/munhyok/gamlendar_selenium/blob/main/core/logs/failedLog.py"
    click notification "https://github.com/munhyok/gamlendar_selenium/blob/main/core/notification/push_notification.py"
    click orchestrator "https://github.com/munhyok/gamlendar_selenium/blob/main/main.py"

    click steamMain "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/steam/steam_main.py"
    click steamPage "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/steam/pageScrap.py"
    click steamScroll "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/steam/scrollScrap.py"
    click steamDetail "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/steam/detailScrap.py"
    click steamBackup "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/steam/backup/"
    click steamLog "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/steam/log/"

    click psMain "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/playstation/playstation_main.py"
    click psPage "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/playstation/pageScrap.py"
    click psDetail "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/playstation/detailScrap.py"
    click psBackup "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/playstation/backup/"
    click psLog "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/playstation/log/"

    click xboxMain "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/xbox/xbox_main.py"
    click xboxPage "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/xbox/pageScrap.py"
    click xboxDetail "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/xbox/detailScrap.py"
    click xboxBackup "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/xbox/backup/"
    click xboxLog "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/xbox/log/"

    click switchMain "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/switch/switch_main.py"
    click switchPage "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/switch/pageScrap.py"
    click switchDetail "https://github.com/munhyok/gamlendar_selenium/blob/main/upcoming/switch/detailScrap.py"
    click switchBackup "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/switch/backup/"
    click switchLog "https://github.com/munhyok/gamlendar_selenium/tree/main/upcoming/switch/log/"

    click testResult "https://github.com/munhyok/gamlendar_selenium/tree/main/test/result/"

    %% Styles
    classDef external fill:#AED9E0,stroke:#333,stroke-width:1px
    classDef scraper fill:#C1E1C1,stroke:#333,stroke-width:1px
    classDef storage fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef core fill:#FDFD96,stroke:#333,stroke-width:1px
    classDef database fill:#FFB347,stroke:#333,stroke-width:1px
    classDef notification fill:#DEA5A4,stroke:#333,stroke-width:1px
```
