# saifayman23/udemy-web-scraping architecture

> Architecture diagram of the GitHub repository saifayman23/udemy-web-scraping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saifayman23/udemy-web-scraping
- Repository: https://github.com/saifayman23/udemy-web-scraping

## Overview

##### Project type and purpose
- This is a **Python-based web scraper** focused on extracting dynamically rendered Udemy course data.
- Purpose:
- Automate browser interaction with Selenium/ChromeDriver
- Extract course details (name, rating, price, etc.)
- Persist raw data (`data.txt`)
- Clean and transform into structured outputs (`output.csv`, `output.xlsx`) using Pandas

##### File structure
- Top-level scripts:
- `scraper.py` – orchestrates Selenium browser, navigates Udemy pages, pulls raw JSON/text, writes `data.txt`
- `cleaner.py` – reads `data.txt`, uses Pandas/Numpy to parse and clean, writes `output.csv` and `output.xlsx`
- Data files:
- `data.txt` – raw scraped dump
- `output.csv`, `output.xlsx` – final tabular exports
- Notebook:
- `udemy_web_scraping.ipynb` – exploratory analysis / demo of scraper + cleaner in a notebook
- Config/resources:
- `LICENSE`, `README.md`, `new.txt` (likely sample or temp data)

##### README for additional insights
- Confirms Selenium + Pandas/Numpy tech stack
- Describes two main phases: scraping and cleaning
- Points to ChromeDriver dependency and command-line execution

a. Main components
- **Selenium WebDriver Layer**
– ChromeDriver + browser instance
– Navigates to Udemy course pages
- **Scraper Module (`scraper.py`)**
– Implements navigation logic, DOM extraction, JSON parsing
- **Raw Data Store**
– File: `data.txt`
- **Cleaner Module (`cleaner.py`)**
– Reads raw data, applies Pandas/Numpy transformations
- **Structured Data Store**
– Files: `output.csv`, `output.xlsx`
- **Optional Notebook**
– `udemy_web_scraping.ipynb` for prototyping or demonstration

b. Relationships and data flow
1. **Start** → invoke `scraper.py`
2. `scraper.py` → initializes Selenium WebDriver (ChromeDriver)
3. WebDriver ↔ Udemy website (HTTP/S requests, JS rendering)
4. Extracted data → write to `data.txt`
5. Invoke `cleaner.py` → read `data.txt` → Pandas/Numpy transformations
6. Cleaned Data → write to `output.csv` / `output.xlsx`
7. (Optional) Notebook ingests `data.txt` / final outputs for analysis

c. Architectural patterns and design principles
- **Separation of concerns**: scraping logic vs. cleaning logic
- **Pipeline pattern**: linear data flow from fetch → raw store → transform → final store
- **Modularity**: two independent scripts that can be chained or run separately

d. Key technologies/frameworks
- **Selenium** + **ChromeDriver** (dynamic JS rendering)
- **Pandas**, **NumPy** (data manipulation)
- **Python 3.9+**

##### Final recommendations
- Keep labels concise and precise (e.g., “scraper.py”, “data.txt”)
- Every file or module that plays a role in the pipeline
- Review with stakeholders to confirm no hidden components (e.g., ChromeDriver path config)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Scraper Phase
    subgraph "Scraper Phase"
        SCRAPER["scraper.py"]:::scrape
        WEBDRIVER["Selenium WebDriver"]:::scrape
    end
    SCRAPER -->|"init"| WEBDRIVER
    WEBDRIVER -->|"HTTP/S, JS"| UDEMY["Udemy Website"]:::external
    UDEMY -->|"rendered content"| WEBDRIVER
    SCRAPER -->|"writes UTF-8 text"| RAW[(data.txt)]:::store

    %% Cleaner Phase
    subgraph "Cleaner Phase"
        CLEANER["cleaner.py"]:::process
    end
    RAW -->|"read text"| CLEANER
    CLEANER -->|"writes CSV/XLSX"| OUTCSV[(output.csv)]:::store
    CLEANER -->|"writes CSV/XLSX"| OUTXLSX[(output.xlsx)]:::store

    %% Notebook Alternate Entry
    NOTEBOOK["udemy_web_scraping.ipynb"]:::external
    NOTEBOOK -->|"reads/writes"| RAW
    NOTEBOOK -->|"reads"| OUTCSV
    NOTEBOOK -->|"reads"| OUTXLSX

    %% Documentation & Config
    subgraph "Documentation & Config"
        README["README.md"]:::external
        LICENSE["LICENSE"]:::external
    end

    %% Click Events
    click SCRAPER "https://github.com/saifayman23/udemy-web-scraping/blob/main/scraper.py"
    click RAW "https://github.com/saifayman23/udemy-web-scraping/blob/main/data.txt"
    click CLEANER "https://github.com/saifayman23/udemy-web-scraping/blob/main/cleaner.py"
    click OUTCSV "https://github.com/saifayman23/udemy-web-scraping/blob/main/output.csv"
    click OUTXLSX "https://github.com/saifayman23/udemy-web-scraping/blob/main/output.xlsx"
    click NOTEBOOK "https://github.com/saifayman23/udemy-web-scraping/blob/main/udemy_web_scraping.ipynb"
    click README "https://github.com/saifayman23/udemy-web-scraping/blob/main/README.md"
    click LICENSE "https://github.com/saifayman23/udemy-web-scraping/tree/main/LICENSE"

    %% Styles
    classDef scrape fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef process fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef store fill:#D3D3D3,stroke:#000,stroke-width:1px;
    classDef external fill:#F5F5F5,stroke:#000,stroke-width:1px,stroke-dasharray:5 5;
```
