How saifayman23/udemy-web-scraping works
Project type and purpose
• This is a Python-based web scraper focused on extracting dynamically rendered Udemy course data.
• Purpose:
• Automate browser interaction with Selenium/ChromeDriver
• Extract course details (name, rating, price, etc.)
• Persist raw data (data.txt)
• Clean and transform into structured outputs (output.csv, output.xlsx) using Pandas
File structure
• Top-level scripts:
• scraper.py – orchestrates Selenium browser, navigates Udemy pages, pulls raw JSON/text, writes data.txt
• cleaner.py – reads data.txt, uses Pandas/Numpy to parse and clean, writes output.csv and output.xlsx
• Data files:
• data.txt – raw scraped dump
• output.csv, output.xlsx – final tabular exports
• Notebook:
• udemy_web_scraping.ipynb – exploratory analysis / demo of scraper + cleaner in a notebook
• Config/resources:
• LICENSE, README.md, new.txt (likely sample or temp data)
README for additional insights
• Confirms Selenium + Pandas/Numpy tech stack
• Describes two main phases: scraping and cleaning
• Points to ChromeDriver dependency and command-line execution
a. Main components
• Selenium WebDriver Layer
– ChromeDriver + browser instance
– Navigates to Udemy course pages
• Scraper Module (scraper.py)
– Implements navigation logic, DOM extraction, JSON parsing
• Raw Data Store
– File: data.txt
• Cleaner Module (cleaner.py)
– Reads raw data, applies Pandas/Numpy transformations
• Structured Data Store
– Files: output.csv, output.xlsx
• Optional Notebook
– udemy_web_scraping.ipynb for prototyping or demonstration
b. Relationships and data flow
1. Start → invoke scraper.py
2. scraper.py → initializes Selenium WebDriver (ChromeDriver)
3. WebDriver ↔ Udemy website (HTTP/S requests, JS rendering)
4. Extracted data → write to data.txt
5. Invoke cleaner.py → read data.txt → Pandas/Numpy transformations
6. Cleaned Data → write to output.csv / output.xlsx
7. (Optional) Notebook ingests data.txt / final outputs for analysis
c. Architectural patterns and design principles
• Separation of concerns: scraping logic vs. cleaning logic
• Pipeline pattern: linear data flow from fetch → raw store → transform → final store
• Modularity: two independent scripts that can be chained or run separately
d. Key technologies/frameworks
• Selenium + ChromeDriver (dynamic JS rendering)
• Pandas, NumPy (data manipulation)
• Python 3.9+
Final recommendations
• Keep labels concise and precise (e.g., “scraper.py”, “data.txt”)
• Every file or module that plays a role in the pipeline
• Review with stakeholders to confirm no hidden components (e.g., ChromeDriver path config)