rishabhvaibhav/assessment_priceeasy

Repository Scope

This repository contains two independent Python web-scraping projects: a Books to Scrape exporter and a Casey’s store-location scraper. There is no shared package or unified runtime; each project owns its own entry point, scraping target, and CSV outputs.

Books to Scrape Script

Project 1/Book_to_scrape/Scrape_books.py is a standalone scraper for the Books to Scrape site. It produces the checked-in dataset at Project 1/Book_to_scrape/books.csv. Its architecture is a direct fetch/parse/export pipeline rather than a reusable application.

Casey’s Modular Application

The primary structured application is under Project 2/caseys_scraper/Prod_debug. Its executable entry point is Project 2/caseys_scraper/Prod_debug/main.py, which coordinates configuration, request execution, extraction, persistence, and run reporting.

Key runtime inputs are Project 2/caseys_scraper/Prod_debug/config.json and Project 2/caseys_scraper/Prod_debug/requirements.txt.

Configuration and Request Boundary

Project 2/caseys_scraper/Prod_debug/core/config.py loads and exposes scraper settings from Project 2/caseys_scraper/Prod_debug/config.json. Proxy material is maintained separately in Project 2/caseys_scraper/Prod_debug/proxies/proxies.txt.

Project 2/caseys_scraper/Prod_debug/core/scraper.py forms the network boundary: it is responsible for obtaining Casey’s source content, likely applying configured request behavior and proxy selection. The external service is Casey’s store-location/web data source; access reliability is an architectural concern because proxy support is explicitly present.

Extraction Stage

Project 2/caseys_scraper/Prod_debug/core/extractor.py separates parsing and normalization of store records from HTTP retrieval. This boundary lets scraper behavior change independently from the expected store-data schema and makes extracted records suitable for multiple output formats.

Storage, State, and Outputs

Project 2/caseys_scraper/Prod_debug/core/storage.py persists extracted data and scraper state. The application produces:

• Project 2/caseys_scraper/Prod_debug/output/caseys_stores_data.csv

• Project 2/caseys_scraper/Prod_debug/output/caseys_stores_data.json

• Project 2/caseys_scraper/Prod_debug/output/scraped_state.json

• Project 2/caseys_scraper/Prod_debug/output/run_summary.json

The state file indicates the scraper supports progress tracking, resumption, deduplication, or incremental execution. The run summary is a separate operational-reporting artifact rather than primary domain data.

Alternate Casey’s Implementation

Project 2/caseys_scraper/Single Script/caseys_scraper.py is a separate monolithic implementation of the same Casey’s scraping concern. It writes Project 2/caseys_scraper/Single Script/caseys_stores_data.csv and should be treated as an alternative execution path, not a module imported by the modular application.

Primary Control Flow

For the modular Casey’s scraper, control flows from main.py through configuration loading, optional proxy/input setup, network scraping, record extraction, and storage/reporting. Data crosses three important boundaries: remote Casey’s content into core/scraper.py, unstructured response data into normalized records in core/extractor.py, and normalized records/state into durable CSV, JSON, and status artifacts via core/storage.py.