# kayden-vs/flipkart-scraper architecture

> Architecture diagram of the GitHub repository kayden-vs/flipkart-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kayden-vs/flipkart-scraper
- Repository: https://github.com/kayden-vs/flipkart-scraper

## Overview

##### Project type and purpose
- This is a Python-based, Scrapy-driven web-scraping tool (“Flipkart Price Tracker”) whose purpose is to crawl Flipkart pages, extract discounted products, verify them against PriceHistory.app, and push Telegram notifications for all-time-low deals.

##### File structure
- Top-level files:
- scrapy.cfg – Scrapy launch configuration
- requirements.txt – Python dependencies
- README.md, LICENSE, .gitignore
- Project package:
flipkart/flipkart/
- spiders/ – spider modules (products.py, search_terms.py, telegram_utils.py)
- middlewares.py – request middleware (UA rotation, headers)
- pipelines.py – item processing placeholder
- settings.py – Scrapy settings (delays, retries, extensions)
- items.py – item definitions
- No persistent database; data flows through pipelines or JSON outputs.

##### README for additional insights
- Architecture section describes each module’s responsibility.
- Dependencies: scrapy, python-dotenv, fake-useragent, HTTP client for PriceHistory API, Telegram Bot API.
- Extensibility points: custom middlewares, pipelines, search terms, future proxy support, Dockerization.

##### Detailed capturing of architectural elements
- Plugin points: where additional middlewares or pipelines plug in.
- Failure-handling layers (retries in settings, error handling in spiders).
- Environment at the top or bottom: “.env → settings.py → modules”.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    Config["Configuration Loader\n(.env → settings.py)"]:::config
    click Config "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/settings.py"

    %% Core Scrapy Components
    Engine["Scrapy Engine"]:::core
    click Engine "https://github.com/kayden-vs/flipkart-scraper/blob/main/scrapy.cfg"

    Scheduler["Scheduler & Downloader"]:::core
    click Scheduler "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/settings.py"

    %% Custom Modules
    Middlewares["Downloader Middlewares\n(UA rotation, rate limiting)"]:::custom
    click Middlewares "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/middlewares.py"

    subgraph "Spiders"
        direction TB
        ProductsSpider["Products Spider\n(core crawl, parse,\nPriceHistory verification)"]:::custom
        click ProductsSpider "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/spiders/products.py"
        SearchTerms["Search Terms Module"]:::custom
        click SearchTerms "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/spiders/search_terms.py"
        TelegramUtils["Telegram Utils\n(notification sender)"]:::custom
        click TelegramUtils "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/spiders/telegram_utils.py"
    end

    ItemsDef["Item Definitions\n(schema)"]:::custom
    click ItemsDef "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/items.py"

    Pipeline["Item Pipeline\n(post-processing, JSON export)"]:::custom
    click Pipeline "https://github.com/kayden-vs/flipkart-scraper/blob/main/flipkart/flipkart/pipelines.py"

    %% External Services
    FlipkartExt["External: Flipkart Website"]:::external
    PriceHistoryExt["External: PriceHistory.app API"]:::external
    TelegramExt["External: Telegram Bot API"]:::external

    %% Data Flows
    Config -->|loads env vars| Engine
    Engine -->|orchestrates| Scheduler
    Scheduler -->|sends requests| FlipkartExt
    Scheduler -->|passes via| Middlewares
    Middlewares -->|inject headers, handle retries| ProductsSpider
    SearchTerms -->|provides keywords| ProductsSpider
    ProductsSpider -->|uses schema| ItemsDef
    ProductsSpider -->|Verify price low| PriceHistoryExt
    ProductsSpider -->|format & send| TelegramUtils
    ProductsSpider -->|"push items"| Pipeline
    Pipeline -->|export JSON| Output[Output Files]

   %% Click Events for external if needed (none)

    %% Styles
    classDef core fill:#ffe4b5,stroke:#ccaa88,stroke-width:1px;
    classDef custom fill:#e0ffff,stroke:#88bbbb,stroke-width:1px;
    classDef external fill:#ffebcd,stroke:#ddbb88,stroke-width:1px;
    classDef config fill:#f0fff0,stroke:#88cc88,stroke-width:1px;
```
