# dev-essbee/iplt20-scraper architecture

> Architecture diagram of the GitHub repository dev-essbee/iplt20-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dev-essbee/iplt20-scraper
- Repository: https://github.com/dev-essbee/iplt20-scraper

## Overview

##### Project type and purpose
- This is a command-line web-scraping tool written in Python.
- Its sole responsibility is to fetch various IPL (Indian Premier League) statistics pages from https://www.iplt20.com, parse the HTML, and export the results as CSV files.

##### File structure
- Top level:
– scraper.py
– Result-Most-Runs-2019.csv (example output)
– README.md, LICENSE, .gitignore
- scraper.py contains all of the logic: CLI interaction, HTTP requests, HTML parsing, data transformation, and CSV generation. There are no sub-directories or separate modules.

##### README for additional insights
- Dependencies: requests, BeautifulSoup4, pandas, numpy, PyInquirer
- Usage: run scraper.py, choose a data set, and script writes CSV files to the current directory.

a. CLI Module
– Uses PyInquirer to present a menu of statistic categories (e.g., Most Runs, Most Wickets).
– Captures user selection.
b. HTTP Client
– Uses the requests library to fetch the HTML page corresponding to the chosen statistic and year.
c. HTML Parser
– Uses BeautifulSoup to navigate the DOM, extract table rows, headings, and numeric data.
d. Data Processor
– Uses pandas (and numpy for any numeric operations) to structure raw lists into DataFrame, perform cleaning or type conversion.
e. CSV Writer
– Leverages pandas’ to_csv() to persist the DataFrame to a .csv file in the working directory.
f. External Service
– The target website (https://www.iplt20.com).

##### Relationships and data flow
- User → CLI Module: triggers the scrape job.
- CLI → HTTP Client: builds request URL, sends GET.
- HTTP Client → HTML Parser: hands off raw HTML.
- HTML Parser → Data Processor: provides raw lists/dicts.
- Data Processor → CSV Writer: hands off structured DataFrame.
- CSV Writer → Local File System: writes Result-*.csv.

##### Architectural patterns and design principles
- Pipeline pattern: each stage (CLI → Fetch → Parse → Process → Persist) forms a sequential pipeline.
- Single-responsibility: although combined in one script, each logical block has its own responsibility.
- External dependency inversion: requests & BeautifulSoup hide HTTP & DOM details, pandas abstracts CSV I/O.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External user
    User["User"]:::external

    %% Main scraper container
    subgraph "scraper.py"
        direction TB
        CLI["CLI Module<br/>(PyInquirer)"]:::module
        HTTP["HTTP Client<br/>(requests)"]:::module
        Parser["HTML Parser<br/>(BeautifulSoup)"]:::module
        Processor["Data Processor<br/>(pandas, numpy)"]:::module
        CSVW["CSV Writer<br/>(pandas.to_csv)"]:::module
    end

    %% External services and storage
    IPLT["IPLT20.com<br/>API/HTML"]:::external
    FS["Local File System<br/>(CSV Output)"]:::storage

    %% Data flow
    User -->|runs script| CLI
    CLI -->|builds URL and GET| HTTP
    HTTP -->|GET request| IPLT
    IPLT -->|HTML response| HTTP
    HTTP -->|raw HTML| Parser
    Parser -->|raw lists/dicts| Processor
    Processor -->|DataFrame| CSVW
    CSVW -->|writes CSV| FS

    %% Click events
    click CLI "https://github.com/dev-essbee/iplt20-scraper/blob/master/scraper.py"
    click HTTP "https://github.com/dev-essbee/iplt20-scraper/blob/master/scraper.py"
    click Parser "https://github.com/dev-essbee/iplt20-scraper/blob/master/scraper.py"
    click Processor "https://github.com/dev-essbee/iplt20-scraper/blob/master/scraper.py"
    click CSVW "https://github.com/dev-essbee/iplt20-scraper/blob/master/scraper.py"

    %% Styles
    classDef module fill:#ADD8E6,stroke:#0000FF,color:#000
    classDef external fill:#F0F8FF,stroke:#4682B4,color:#000
    classDef storage fill:#CCFFCC,stroke:#008000,color:#000
```
