# ibrahim-string/sitescraper architecture

> Architecture diagram of the GitHub repository ibrahim-string/sitescraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibrahim-string/sitescraper
- Repository: https://github.com/ibrahim-string/sitescraper

## Overview

##### Project Type and Purpose
- This is an open-source Python tool for web scraping, focused on extracting YouTube video details, comments, and links.
- The README clearly outlines three primary methods that use Selenium (with Firefox and Chrome web drivers) to interact with YouTube pages and then package the scraped data into a dictionary or Pandas DataFrame.
- It’s a lightweight module intended to be integrated into user scripts or executed in environments like Google Colab.

##### Analyzing the File Structure
- Top-level files include configuration files (setup.py, setup.cfg), licensing, and distribution archives, which shows that this project is designed as a distributable Python package.
- The core functionality lives in the “SiteScraper” directory (with SiteScraper.py and __init__.py) which is the main module exposed to users.
- Notice the presence of egg-info, which is typical for Python packages, suggesting standard packaging practices.

##### Key System Components
a. User/Application Layer:
- Represents the client code that imports the SiteScraper module and calls its methods.

b. SiteScraper Module:
- Contains the core logic for each functionality:
- yt_vedio_data() for scraping video details (title, views, upload time).
- yt_vedio_comment() for scraping video comments.
- yt_vedio_links() for extracting video links.
- This is encapsulated in the SiteScraper.py file.

c. Selenium Web Driver Components:
- There are two web driver implementations:
- Firefox (via geckodriver-autoinstaller) used by yt_vedio_data() and yt_vedio_comment().
- Chrome web driver (installed via the Google Colab snippet) used by yt_vedio_links().
- These drivers are responsible for automating the browser interactions with YouTube.

d. External Data Sources:
- YouTube’s web pages which are the targets for scraping.

e. Data Handling and Transformation:
- Use of Pandas for converting raw scraped data into a DataFrame format (especially in the yt_vedio_comment() and yt_vedio_links() methods).

f. Environment/Deployment:
- Instructions for running in Google Colab and the use of Linux package management for setting up the proper drivers.

##### Relationships and Interactions
- User Code → SiteScraper Module:
- The user imports the module and invokes specific scraping methods.

- SiteScraper Module → Selenium Web Drivers:
- Depending on the method used, SiteScraper creates an instance of a Firefox or Chrome web driver.
- The module sends requests to load YouTube URLs.

- Selenium Web Drivers → YouTube Website:
- Web drivers simulate human browsing, loading the webpage, and allow extraction of DOM elements.

- Data Flow Back:
- Extracted data is processed (e.g., parsed into dictionaries or DataFrame objects) and then returned to the user code for further use (like saving to CSV).

##### Architectural Patterns and Design Principles
- Modular Design:
- The project encapsulates its primary functionalities within a single module, making it easier to extend (as noted in “New methods will be commited for twitter and reddit”).

- Dependency Management:
- External libraries such as Selenium, Pandas, and geckodriver-autoinstaller are specified and installed via pip.

- Separation of Concerns:
- The scraping logic (data extraction) is distinct from data transformation (packing into DataFrames/dictionaries) and from the environment setup (detailed instructions for Google Colab).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User/Application Layer"
        user["User/Application Code"]:::user
    end

    subgraph "Core Functionality"
        mod["SiteScraper Module"]:::module
        yt1["yt_vedio_data()"]:::method
        yt2["yt_vedio_comment()"]:::method
        yt3["yt_vedio_links()"]:::method
        mod --> yt1
        mod --> yt2
        mod --> yt3
    end

    subgraph "Web Drivers"
        firefox["Selenium Firefox Web Driver (geckodriver-autoinstaller)"]:::selenium
        chrome["Selenium Chrome Web Driver"]:::selenium
    end

    subgraph "External Websites"
        yt["YouTube Website"]:::external
    end

    subgraph "Data Transformation"
        panda["Data Processing / Pandas Transformation"]:::data
    end

    subgraph "Environment & Packaging"
        pkg["Packaging and Environment Setup"]:::packaging
    end

    user -->|"imports"| mod
    yt1 -->|"triggers"| firefox
    yt2 -->|"triggers"| firefox
    yt3 -->|"triggers"| chrome
    firefox -->|"loads"| yt
    chrome -->|"loads"| yt
    firefox -->|"returns"| mod
    chrome -->|"returns"| mod
    mod -->|"processes"| panda
    panda -->|"sendsOutput"| user
    pkg -->|"configures"| mod

    classDef user fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef module fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef method fill:#e0ffe0,stroke:#339933,stroke-width:2px;
    classDef selenium fill:#ccccff,stroke:#0000cc,stroke-width:2px;
    classDef external fill:#ffffcc,stroke:#cccc00,stroke-width:2px;
    classDef data fill:#ccffff,stroke:#00cccc,stroke-width:2px;
    classDef packaging fill:#ffebcc,stroke:#cc6600,stroke-width:2px;

    click user "https://github.com/ibrahim-string/sitescraper/blob/master/README.md"
    click mod "https://github.com/ibrahim-string/sitescraper/blob/master/SiteScraper/SiteScraper.py"
    click mod "https://github.com/ibrahim-string/sitescraper/blob/master/SiteScraper/__init__.py"
    click firefox "https://github.com/ibrahim-string/sitescraper/blob/master/setup.py"
    click chrome "https://github.com/ibrahim-string/sitescraper/blob/master/SiteScraper.egg-info"
    click panda "https://github.com/ibrahim-string/sitescraper/blob/master/SiteScraper/SiteScraper.py"
    click pkg "https://github.com/ibrahim-string/sitescraper/blob/master/setup.py"
    click pkg "https://github.com/ibrahim-string/sitescraper/blob/master/setup.cfg"
```
