# chafic123/ebay-global-tech-scraping architecture

> Architecture diagram of the GitHub repository chafic123/ebay-global-tech-scraping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chafic123/ebay-global-tech-scraping
- Repository: https://github.com/chafic123/ebay-global-tech-scraping

## Overview

##### Main Components
- Data Scraping Module: Represented by the scraper (scraper.py) that uses tools like Selenium, webdriver_manager, and fake_useragent to fetch tech deals data from eBay. This component outputs a CSV file (ebay_tech_deals.csv).
- Data Cleaning & Processing Module: Represented by clean_data.py, which takes the scraped CSV file, performs cleaning (removing currency symbols, handling missing values, etc.), and outputs a cleaned CSV (cleaned_ebay_deals.csv).
- Exploratory Data Analysis (EDA) and Visualization Module: Represented by the Jupyter Notebook (EDA.ipynb seen through the README references and related checkpoints) where analysis, visual charts, and statistical evaluations (using libraries such as pandas, numpy, matplotlib, and seaborn) are performed.
- Automation & CI/CD Module: Represented by the GitHub Actions workflows (in .github/workflows/scraper.yml) that are set to trigger the data scraping, subsequent cleaning, and updating of the analysis automatically on a schedule.
- Supporting Assets: This includes the CSV files produced (ebay_tech_deals.csv and cleaned_ebay_deals.csv) that serve as data artifacts between the modules.

##### Relationships & Data Flow
- The Data Scraping Module pulls data from eBay and generates raw CSV output.
- The raw CSV feeds into the Data Cleaning Module which processes the data and generates a cleaned dataset.
- The cleaned data is then consumed by the EDA/Visualization Module for analysis and creation of visual insights.
- The GitHub Actions automation ensures that each of these steps is run as per the scheduled workflow, thereby updating the data artifacts and results automatically.

##### Architectural Patterns and Technologies
- Although the project is not a traditional full-stack application, treat each phase (scraping, cleaning, analysis) as separate layers. This modular layering simplifies adding new features or modifications.
- CI/CD automation provided by GitHub Actions which orchestrates the data fetching to insight generation process.
– Python and its libraries (pandas, numpy, matplotlib, seaborn)
– Selenium for automation of web scraping
– Jupyter Notebook environment for EDA
– GitHub Actions for scheduling and automation

##### Guiding Principles
– Rectangles for scripts/modules (Scraper, Cleaner)
- Be explicit by including the CSV intermediary files as stored data that passes from one module to another.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Automation & CI/CD (GitHub Actions)"]:::automation
    B["Data Scraper (scraper.py)"]:::module
    C["Raw CSV (ebay_tech_deals.csv)"]:::file
    D["Data Cleaner (clean_data.py)"]:::module
    E["Clean CSV (cleaned_ebay_deals.csv)"]:::file
    F["EDA & Visualization (ebay_cleaned_data_Visual.ipynb)"]:::notebook
    G["Notebook Checkpoint (.ipynb_checkpoints/ebay_cleaned_data_Visual-checkpoint.ipynb)"]:::file

    A -->|"triggers"| B
    B -->|"outputs"| C
    C -->|"feeds"| D
    D -->|"produces"| E
    E -->|"consumed_by"| F
    F -->|"references"| G

    click A "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/.github/workflows/scraper.yml"
    click B "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/scraper.py"
    click C "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/ebay_tech_deals.csv"
    click D "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/clean_data.py"
    click E "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/cleaned_ebay_deals.csv"
    click F "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/ebay_cleaned_data_Visual.ipynb"
    click G "https://github.com/chafic123/ebay-global-tech-scraping/blob/main/.ipynb_checkpoints/ebay_cleaned_data_Visual-checkpoint.ipynb"

    classDef module fill:#b3e5fc,stroke:#01579b,stroke-width:2px;
    classDef file fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px;
    classDef automation fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px;
    classDef notebook fill:#ffecb3,stroke:#ff9800,stroke-width:2px;
```
