How dev-essbee/iplt20-scraper works
Project type and purpose
• This is a command-line web-scraping tool written in Python.
• Its sole responsibility is to fetch various IPL (Indian Premier League) statistics pages from https://www.iplt20.com, parse the HTML, and export the results as CSV files.
File structure
• Top level:
– scraper.py
– Result-Most-Runs-2019.csv (example output)
– README.md, LICENSE, .gitignore
• scraper.py contains all of the logic: CLI interaction, HTTP requests, HTML parsing, data transformation, and CSV generation. There are no sub-directories or separate modules.
README for additional insights
• Dependencies: requests, BeautifulSoup4, pandas, numpy, PyInquirer
• Usage: run scraper.py, choose a data set, and script writes CSV files to the current directory.
a. CLI Module
– Uses PyInquirer to present a menu of statistic categories (e.g., Most Runs, Most Wickets).
– Captures user selection.
b. HTTP Client
– Uses the requests library to fetch the HTML page corresponding to the chosen statistic and year.
c. HTML Parser
– Uses BeautifulSoup to navigate the DOM, extract table rows, headings, and numeric data.
d. Data Processor
– Uses pandas (and numpy for any numeric operations) to structure raw lists into DataFrame, perform cleaning or type conversion.
e. CSV Writer
– Leverages pandas’ to_csv() to persist the DataFrame to a .csv file in the working directory.
f. External Service
– The target website (https://www.iplt20.com).
Relationships and data flow
• User → CLI Module: triggers the scrape job.
• CLI → HTTP Client: builds request URL, sends GET.
• HTTP Client → HTML Parser: hands off raw HTML.
• HTML Parser → Data Processor: provides raw lists/dicts.
• Data Processor → CSV Writer: hands off structured DataFrame.
• CSV Writer → Local File System: writes Result-*.csv.
Architectural patterns and design principles
• Pipeline pattern: each stage (CLI → Fetch → Parse → Process → Persist) forms a sequential pipeline.
• Single-responsibility: although combined in one script, each logical block has its own responsibility.
• External dependency inversion: requests & BeautifulSoup hide HTTP & DOM details, pandas abstracts CSV I/O.