# sudhanshu1402/technical_internship_program architecture

> Architecture diagram of the GitHub repository sudhanshu1402/technical_internship_program, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sudhanshu1402/technical_internship_program
- Repository: https://github.com/sudhanshu1402/technical_internship_program

## Overview

##### Project type and purpose
- This is a monorepo of Python “web‑scraping” (crawler) projects built with Scrapy.
- Each top‑level folder (EXTRA_WEBSITES, RETAIL_WEBSITES, TYRES) groups one or more Scrapy spiders that extract structured data from specific target sites and persist it in CSV, Excel, SQL or MongoDB.

##### File‐structure analysis
- Top‑level: three domain folders + a global README.
- Under each domain folder: one subfolder per target website.
- Each subfolder contains:
– items.py (data schema)
– spider module(s) (*.py)
– pipelines.py (post‑processing & storage)
– middlewares.py (request/response hooks)
– settings.py (crawler configuration)
– sometimes CSV or DOCX storage templates
- No centralized scheduler: each spider group is self‑contained.

##### README insights
- Uses Python with Scrapy framework plus optional BeautifulSoup, Selenium.
- Storage via CSV, Excel, MongoDB, SQL.
- Main scraping methods: HTML/DOM/XPath parsing.

a. Main components
1. Scheduler/CLI (developer or cron job triggers)
2. Scrapy Engine
- Downloader Middleware
- Scheduler (internal queue)
3. Spider Modules (one per site)
4. Item Pipeline
5. Storage Adapters:
– CSV/Excel writer
– MongoDB client
– SQL client
6. External Websites (targets)
b. Relationships & data flow
1. Scheduler → Scrapy Engine (start requests)
2. Engine → Downloader Middleware → HTTP(s) requests → External Website
3. External Website → HTTP response → Downloader Middleware → Spider parse()
4. Spider parse() → yields Items → Engine → Item Pipeline
5. Pipeline processes items → hands off to Storage Adapter(s)
6. Storage Adapter writes to CSV/Excel/DB
c. Architectural patterns/design principles
- Scrapy’s event‑driven crawler architecture (engine + spider callbacks + pipeline)
- Modular spider design: each site lives in its own submodule for isolation
- Middleware pipeline pattern for request/response customization
- Item pipeline pattern for sequential data transformation and output
d. Key technologies/frameworks
- Python 3.x, Scrapy framework
- Optional: BeautifulSoup, Selenium for dynamic pages
- Storage: CSV modules, Pandas/Excel writer, PyMongo (MongoDB), SQLAlchemy/pyodbc

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scheduler/CLI Entry Points
    subgraph "Scheduler/CLI"  
        SCHED_PIN["Pincode CLI"]:::custom
        SCHED_MARKET["MarketAir CLI"]:::custom
        SCHED_SUPPLY["SupplyHouse CLI"]:::custom
        SCHED_SUSTAIN["SustainableSupply CLI"]:::custom
    end

    %% Core Framework
    subgraph "Scrapy Engine"  
        EngineCore["Engine Core"]:::core
        IntSched["Internal Scheduler"]:::core
        DownloaderMDW["Downloader Middleware"]:::core
    end

    %% Spider Modules
    subgraph "Spider Modules"  
        ANGEL["ANGELMATCH Spider"]:::custom
        DISTRICT["DISTRICT Spider"]:::custom
        PINCODE["PINCODE Spider"]:::custom
        AFSUPPLY["AFSUPPLY Spider"]:::custom
        MARKETAIR["MARKETAIR Spider"]:::custom
        SUPPLYHOUSE["SUPPLYHOUSE Spider"]:::custom
        SUSTSUPPLY["SUSTAINABLESUPPLY Spider"]:::custom
        APOLLO["APOLLO Tyres Spider"]:::custom
        BIRLA["BIRLA Tyres Spider"]:::custom
        BRIDGE["BRIDGESTONE Tyres Spider"]:::custom
    end

    %% Item Pipeline
    Pipeline["Item Pipeline"]:::custom

    %% Storage Adapters
    CSV["CSV/Excel Writer"]:::storage
    MongoDB["MongoDB Client"]:::storage
    SQLDB["SQL Client"]:::storage

    %% External Websites
    subgraph "External Websites"  
        EXTSITE["Target Website"]:::external
    end

    %% Connections
    SCHED_PIN -->|starts| EngineCore
    SCHED_MARKET -->|starts| EngineCore
    SCHED_SUPPLY -->|starts| EngineCore
    SCHED_SUSTAIN -->|starts| EngineCore

    EngineCore -->|queues| IntSched
    IntSched -->|requests| DownloaderMDW
    DownloaderMDW -->|HTTP Request| EXTSITE
    EXTSITE -->|HTTP Response| DownloaderMDW
    DownloaderMDW -->|parses| ANGEL
    DownloaderMDW -->|parses| DISTRICT
    DownloaderMDW -->|parses| PINCODE
    DownloaderMDW -->|parses| AFSUPPLY
    DownloaderMDW -->|parses| MARKETAIR
    DownloaderMDW -->|parses| SUPPLYHOUSE
    DownloaderMDW -->|parses| SUSTSUPPLY
    DownloaderMDW -->|parses| APOLLO
    DownloaderMDW -->|parses| BIRLA
    DownloaderMDW -->|parses| BRIDGE

    ANGEL -->|yields Items| Pipeline
    DISTRICT -->|yields Items| Pipeline
    PINCODE -->|yields Items| Pipeline
    AFSUPPLY -->|yields Items| Pipeline
    MARKETAIR -->|yields Items| Pipeline
    SUPPLYHOUSE -->|yields Items| Pipeline
    SUSTSUPPLY -->|yields Items| Pipeline
    APOLLO -->|yields Items| Pipeline
    BIRLA -->|yields Items| Pipeline
    BRIDGE -->|yields Items| Pipeline

    Pipeline -->|writes| CSV
    Pipeline -->|writes| MongoDB
    Pipeline -->|writes| SQLDB

    %% Click Events - Scheduler/CLI
    click SCHED_PIN "https://github.com/sudhanshu1402/technical_internship_program/blob/main/EXTRA_WEBSITES/PINCODE/main.py"
    click SCHED_MARKET "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/MARKETAIR/main.py"
    click SCHED_SUPPLY "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/SUPPLYHOUSE/main.py"
    click SCHED_SUSTAIN "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/SUSTAINABLESUPPLY/main.py"

    %% Click Events - Spider Modules
    click ANGEL "https://github.com/sudhanshu1402/technical_internship_program/blob/main/EXTRA_WEBSITES/ANGELMATCH/angelmatchspider.py"
    click DISTRICT "https://github.com/sudhanshu1402/technical_internship_program/blob/main/EXTRA_WEBSITES/DISTRICT/districtspider.py"
    click PINCODE "https://github.com/sudhanshu1402/technical_internship_program/blob/main/EXTRA_WEBSITES/PINCODE/pincodespider.py"
    click AFSUPPLY "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/AFSUPPLY/afsupplyspider.py"
    click MARKETAIR "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/MARKETAIR/marketairspider.py"
    click SUPPLYHOUSE "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/SUPPLYHOUSE/supplyhousespider.py"
    click SUSTSUPPLY "https://github.com/sudhanshu1402/technical_internship_program/blob/main/RETAIL_WEBSITES/SUSTAINABLESUPPLY/sustainablesupplyspider.py"
    click APOLLO "https://github.com/sudhanshu1402/technical_internship_program/blob/main/TYRES/APOLLO/apollotyresspider.py"
    click BIRLA "https://github.com/sudhanshu1402/technical_internship_program/blob/main/TYRES/BIRLA/birlatyresspider.py"
    click BRIDGE "https://github.com/sudhanshu1402/technical_internship_program/blob/main/TYRES/BRIDGESTONE/bridgestonetyresspider.py"

    %% Click Events - Storage Templates
    click DISTRICT "https://github.com/sudhanshu1402/technical_internship_program/blob/main/EXTRA_WEBSITES/DISTRICT/dist.csv"
    click PINCODE "https://github.com/sudhanshu1402/technical_internship_program/blob/main/EXTRA_WEBSITES/PINCODE/pincodes.csv"

    %% Styles
    classDef core fill:#cceeff,stroke:#3399ff
    classDef custom fill:#ccffcc,stroke:#33cc33
    classDef storage fill:#ffddaa,stroke:#ff8800
    classDef external fill:#dddddd,stroke:#888888,stroke-dasharray: 5 5
```
