How kayden-vs/flipkart-scraper works
Project type and purpose
• This is a Python-based, Scrapy-driven web-scraping tool (“Flipkart Price Tracker”) whose purpose is to crawl Flipkart pages, extract discounted products, verify them against PriceHistory.app, and push Telegram notifications for all-time-low deals.
File structure
• Top-level files:
• scrapy.cfg – Scrapy launch configuration
• requirements.txt – Python dependencies
• README.md, LICENSE, .gitignore
• Project package:
flipkart/flipkart/
• spiders/ – spider modules (products.py, search_terms.py, telegram_utils.py)
• middlewares.py – request middleware (UA rotation, headers)
• pipelines.py – item processing placeholder
• settings.py – Scrapy settings (delays, retries, extensions)
• items.py – item definitions
• No persistent database; data flows through pipelines or JSON outputs.
README for additional insights
• Architecture section describes each module’s responsibility.
• Dependencies: scrapy, python-dotenv, fake-useragent, HTTP client for PriceHistory API, Telegram Bot API.
• Extensibility points: custom middlewares, pipelines, search terms, future proxy support, Dockerization.
Detailed capturing of architectural elements
• Plugin points: where additional middlewares or pipelines plug in.
• Failure-handling layers (retries in settings, error handling in spiders).
• Environment at the top or bottom: “.env → settings.py → modules”.