# skshivammahajan/dream11_scrapping architecture

> Architecture diagram of the GitHub repository skshivammahajan/dream11_scrapping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skshivammahajan/dream11_scrapping
- Repository: https://github.com/skshivammahajan/dream11_scrapping

## Overview

##### Project type and purpose
- This is a Python-based web-scraping/automation tool (not a full-stack web app).
- Its purpose is to (a) log into Dream11 via OTP and persist cookies, and (b) scrape the live leaderboard and individual team details, then export them to an Excel file.

##### File structure
- Top-level files:
- Dream11_login.py – handles user login via Selenium, captures and serializes cookies to cookie.txt
- Dream11_crawler.py – reads cookie.txt, launches a Selenium session to bypass login, navigates to the leaderboard, uses BeautifulSoup to parse HTML, writes to DREAM11.xlsx
- README.md – project overview and instructions
- Key dependencies/config:
- Selenium + ChromeDriver (for browser automation)
- bs4 (for HTML parsing)
- A local file store (cookie.txt, DREAM11.xlsx)
- No nested directories—this is a simple two-script CLI utility.

##### Extract insights from README
- Two-step workflow: login → cookie persistence → reuse cookie for scraping
- Uses real browser (ChromeDriver) so network interactions are visible
- Output: Excel spreadsheet

a. Main components:
- User (CLI)
- Dream11_login.py (Login Module)
- cookie.txt (Local Cookie Store)
- Dream11_crawler.py (Crawler Module)
- Selenium WebDriver & Chrome browser (Automation Engine)
- Dream11 Web Server (External Service)
- BeautifulSoup Parser (HTML → data)
- Excel Writer (Output Module)
b. Relationships & data flows:
1. User invokes Dream11_login.py → Selenium WebDriver launches Chrome → user enters mobile + OTP → Dream11 server authenticates → session cookie captured → cookie.txt written
2. User invokes Dream11_crawler.py → script loads cookie.txt → Selenium WebDriver launches Chrome with injected cookie → browser fetches leaderboard pages from Dream11 server → page HTML returned → BeautifulSoup parses DOM → extracted data sent to Excel Writer → DREAM11.xlsx output
c. Architectural patterns/design principles:
- Separation of Concerns: “Login” vs “Scrape” modules
- Stateless scripts with cookie-based session persistence
- Pipeline: Browser Automation → Parsing → Data Export
d. Key technologies/frameworks:
- Python 3.x, Selenium WebDriver, ChromeDriver, BeautifulSoup (bs4), openpyxl/xlsxwriter (for Excel)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Actions
    User["User (CLI)"]
    
    %% Login Workflow
    User -->|"invoke login"| Login["Dream11_login.py"]
    Login -->|"launch"| WebDriver["Selenium WebDriver"]:::automation
    WebDriver -->|"controls"| Browser["Chrome Browser"]:::automation
    Browser -->|"HTTPS Request"| Server["Dream11 Web Server"]:::external
    Server -->|"HTML + Set-Cookie"| Browser
    Browser -->|"get cookies"| Login
    Login -->|"write file"| CookieFile[(cookie.txt)]:::datastore

    %% Crawler Workflow
    User -->|"invoke crawler"| Crawler["Dream11_crawler.py"]
    Crawler -->|"read file"| CookieFile
    Crawler -->|"launch"| WebDriver
    WebDriver -->|"controls"| Browser
    Crawler -->|"inject cookie"| Browser
    Browser -->|"HTTPS Request"| Server
    Server -->|"HTML Response"| Browser
    Browser -->|"page source"| Crawler
    Crawler -->|"parse"| Parser["BeautifulSoup Parser"]:::parser
    Parser -->|"data frame"| ExcelWriter["Excel Writer"]
    ExcelWriter -->|"write file"| Xlsx[(DREAM11.xlsx)]:::datastore

    %% Documentation
    Documentation["README.md"]
    
    %% Click Events
    click Login "https://github.com/skshivammahajan/dream11_scrapping/blob/master/Dream11_login.py"
    click Crawler "https://github.com/skshivammahajan/dream11_scrapping/blob/master/Dream11_crawler.py"
    click Documentation "https://github.com/skshivammahajan/dream11_scrapping/blob/master/README.md"
    
    %% Styles
    classDef automation fill:#add8e6,stroke:#000,stroke-width:1px
    classDef datastore fill:#90ee90,stroke:#000,stroke-width:1px
    classDef parser fill:#ffa500,stroke:#000,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:1px
```
