How skshivammahajan/dream11_scrapping works
Project type and purpose
• This is a Python-based web-scraping/automation tool (not a full-stack web app).
• Its purpose is to (a) log into Dream11 via OTP and persist cookies, and (b) scrape the live leaderboard and individual team details, then export them to an Excel file.
File structure
• Top-level files:
• Dream11_login.py – handles user login via Selenium, captures and serializes cookies to cookie.txt
• Dream11_crawler.py – reads cookie.txt, launches a Selenium session to bypass login, navigates to the leaderboard, uses BeautifulSoup to parse HTML, writes to DREAM11.xlsx
• README.md – project overview and instructions
• Key dependencies/config:
• Selenium + ChromeDriver (for browser automation)
• bs4 (for HTML parsing)
• A local file store (cookie.txt, DREAM11.xlsx)
• No nested directories—this is a simple two-script CLI utility.
Extract insights from README
• Two-step workflow: login → cookie persistence → reuse cookie for scraping
• Uses real browser (ChromeDriver) so network interactions are visible
• Output: Excel spreadsheet
a. Main components:
• User (CLI)
• Dream11_login.py (Login Module)
• cookie.txt (Local Cookie Store)
• Dream11_crawler.py (Crawler Module)
• Selenium WebDriver & Chrome browser (Automation Engine)
• Dream11 Web Server (External Service)
• BeautifulSoup Parser (HTML → data)
• Excel Writer (Output Module)
b. Relationships & data flows:
1. User invokes Dream11_login.py → Selenium WebDriver launches Chrome → user enters mobile + OTP → Dream11 server authenticates → session cookie captured → cookie.txt written
2. User invokes Dream11_crawler.py → script loads cookie.txt → Selenium WebDriver launches Chrome with injected cookie → browser fetches leaderboard pages from Dream11 server → page HTML returned → BeautifulSoup parses DOM → extracted data sent to Excel Writer → DREAM11.xlsx output
c. Architectural patterns/design principles:
• Separation of Concerns: “Login” vs “Scrape” modules
• Stateless scripts with cookie-based session persistence
• Pipeline: Browser Automation → Parsing → Data Export
d. Key technologies/frameworks:
• Python 3.x, Selenium WebDriver, ChromeDriver, BeautifulSoup (bs4), openpyxl/xlsxwriter (for Excel)