How rmadhok/scrapy-crawl-asp works
Project type and purpose
• This is a Python-based heavy-duty web-crawling framework whose primary goal is to scrape tabular data from an ASP.net site that uses JavaScript postbacks rather than distinct URLs per page. It was originally built on Scrapy (you have the standard Scrapy project layout under eccrawler/eccrawler) but currently falls back to a custom requests-based script (ec_crawler.py and scrape_functions.py) for the actual scraping.
File structure
• Top level:
• ec_crawler.py – standalone script driving the requests-based scraper
• scrape_functions.py – contains reusable functions to extract form fields (VIEWSTATE, EVENTVALIDATION), parse HTML, and paginate
• eccrawler/ – Scrapy project root, containing:
– eccrawler/items.py – item definitions for scraped fields
– eccrawler/pipelines.py – pipeline to write items to CSV
– eccrawler/settings.py – Scrapy settings (concurrency, user-agent, pipeline activation)
– eccrawler/spiders/ec_spider.py – Scrapy spider encapsulating the crawl logic
• scrapy.cfg – Scrapy entrypoint config
• test/ – unit tests for the scraping functions (test_scrapy.py)
README for additional insights
• The README explains the JavaScript postback tactic, the form-request approach under Scrapy, and notes the fallback to the requests library when Scrapy is broken. The intended data sink is a CSV file (via the pipeline).
Elements
• Annotations for form-data extraction steps (VIEWSTATE/EVENTVALIDATION).