How munhyok/gamlendar_selenium works
Project Type and Purpose
• This is a Python-based ETL scraping tool (“Gamlendar Selenium”) that automatically collects upcoming game release data from multiple platforms (Steam, Playstation, Xbox, Switch), cleans the raw data, backs it up as CSVs, logs failures, and loads the cleaned data into MariaDB (and then to MongoDB). It uses Selenium (via a WebDriver singleton) for data collection.
Top-Level Components
• External Services / Data Sources (Steam, Playstation, Xbox, Switch websites)
• Scraper Modules (one per platform under upcoming/)
• Core Pipeline
• WebDriver manager (core/Webdriver.py)
• Data layer (core/data/…): concatData, csvSearch, dataInsert
• Cleaning layer (core/data_cleaning/DataCleaning.py)
• Database layer (core/database/Database.py, MariaDB + MongoDB sync)
• Logging (core/logs/failedLog.py)
• Notifications (core/notification/push_notification.py)
• File Storage (backup CSV folders under upcoming/…/backup and test/result)
• Databases (MariaDB, MongoDB)
• Orchestration / Entrypoint (main.py, playstation_main.py, steam_main.py, etc.)
• Tests (test/)
Relationships and Data Flow
1. Orchestrator (main.py or each platform main) invokes the WebDriver singleton.
2. The Scraper Module (pageScrap.py → detailScrap.py) uses Selenium to crawl the platform’s upcoming list pages and detail pages on External Services.
3. Raw data is written to File Storage as timestamped CSV backups.
4. On each fetch or in a batch, Core data layer reads the backups:
a. concatData.py merges multiple CSVs
b. csvSearch.py looks up existing records
c. dataInsert.py writes cleaned records into Database layer
5. core/data_cleaning/DataCleaning.py processes strings, dates, filters out adult titles, normalizes names.
6. core/database/Database.py (singleton) handles writing to MariaDB, then optionally syncing to MongoDB.
7. core/logs/failedLog.py catches and persists scraping/processing errors into per-platform log files.
8. core/notification/push_notification.py can send alerts on critical failures or new releases.
Architectural Patterns and Principles
• ETL pipeline pattern: Extract (Selenium scrapers) → Transform (data cleaning) → Load (MariaDB/MongoDB).
• Singleton pattern: WebDriver manager and Database connector use a single instance.
• Separation of concerns: platform-specific scrapers vs. shared core functionality.
• Modular file backup storage for resiliency and replayability.
• Layered architecture: UI (Selenium), Service (core pipeline), Persistence (DB).