# rmadhok/scrapy-crawl-asp architecture

> Architecture diagram of the GitHub repository rmadhok/scrapy-crawl-asp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rmadhok/scrapy-crawl-asp
- Repository: https://github.com/rmadhok/scrapy-crawl-asp

## Overview

##### Project type and purpose
- This is a Python-based heavy-duty web-crawling framework whose primary goal is to scrape tabular data from an ASP​.net site that uses JavaScript postbacks rather than distinct URLs per page. It was originally built on Scrapy (you have the standard Scrapy project layout under `eccrawler/eccrawler`) but currently falls back to a custom requests-based script (`ec_crawler.py` and `scrape_functions.py`) for the actual scraping.

##### File structure
- Top level:
- `ec_crawler.py` – standalone script driving the requests-based scraper
- `scrape_functions.py` – contains reusable functions to extract form fields (VIEWSTATE, EVENTVALIDATION), parse HTML, and paginate
- `eccrawler/` – Scrapy project root, containing:
– `eccrawler/items.py` – item definitions for scraped fields
– `eccrawler/pipelines.py` – pipeline to write items to CSV
– `eccrawler/settings.py` – Scrapy settings (concurrency, user-agent, pipeline activation)
– `eccrawler/spiders/ec_spider.py` – Scrapy spider encapsulating the crawl logic
- `scrapy.cfg` – Scrapy entrypoint config
- `test/` – unit tests for the scraping functions (`test_scrapy.py`)

##### README for additional insights
- The README explains the JavaScript postback tactic, the form-request approach under Scrapy, and notes the fallback to the requests library when Scrapy is broken. The intended data sink is a CSV file (via the pipeline).

##### Elements
- Annotations for form-data extraction steps (VIEWSTATE/EVENTVALIDATION).

## Diagram (Mermaid)

```mermaid
graph TD
    External["ASP.NET Web Server\nenvironmentclearance.nic.in"]:::external

    subgraph "Scrapy Flow"
        Spider["ec_spider.py\n(Spider)"]:::scrapy
        FormReq["FormRequest\n(VIEWSTATE/EVENTVALIDATION)"]:::scrapy
        Items["items.py\n(Item Definitions)"]:::scrapy
        Pipeline["pipelines.py\n(CSV Pipeline)"]:::scrapy
        Settings["settings.py\n(Scrapy Settings)"]:::config
        CFG["scrapy.cfg\n(Configuration)"]:::config
    end

    subgraph "Requests Flow"
        Fallback["ec_crawler.py\n(Fallback Script)"]:::fallback
        Utils["scrape_functions.py\n(HTML Parser & Form Utilities)"]:::fallback
    end

    CSV["output.csv\n(CSV Store)"]:::data
    Test["test_scrapy.py\n(Unit Tests)"]:::test
    Doc["README.md\n(Documentation)"]:::doc
    License["LICENSE.md\n(License)"]:::doc

    %% Connections
    Spider -->|HTTP GET/POST| External
    External -->|HTML w/hidden fields| Spider
    Spider -->|"yields Items"| Items
    Items -->|"pushes to"| Pipeline
    Pipeline -->|"writes rows to"| CSV
    Settings --> Spider
    CFG --> Spider

    Fallback -->|"calls"| Utils
    Utils -->|HTTP GET/POST| External
    External -->|HTML response| Utils
    Utils -->|"parses & writes to"| CSV

    Test -->|"tests parsing"| Utils
    Test -->|"tests spider logic"| Spider

    Doc --> Spider
    Doc --> Fallback
    License --> Doc

    %% Click Events
    click Spider "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/eccrawler/eccrawler/spiders/ec_spider.py"
    click Items "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/eccrawler/eccrawler/items.py"
    click Pipeline "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/eccrawler/eccrawler/pipelines.py"
    click Settings "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/eccrawler/eccrawler/settings.py"
    click CFG "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/eccrawler/scrapy.cfg"
    click Fallback "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/ec_crawler.py"
    click Utils "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/scrape_functions.py"
    click Test "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/test/test_scrapy.py"
    click Doc "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/README.md"
    click License "https://github.com/rmadhok/scrapy-crawl-asp/blob/master/LICENSE.md"

    %% Styles
    classDef scrapy fill:#ccffcc,stroke:#2d662d,color:#000;
    classDef fallback fill:#ffe0b3,stroke:#b37400,color:#000;
    classDef data fill:#ccccff,stroke:#333399,color:#000;
    classDef external fill:#e0e0e0,stroke:#666666,color:#000;
    classDef config fill:#cce5ff,stroke:#2d4e8f,color:#000;
    classDef test fill:#ffcccc,stroke:#993333,color:#000;
    classDef doc fill:#f2f2f2,stroke:#999999,color:#000;
```
