# rishabhvaibhav/assessment_priceeasy architecture

> Architecture diagram of the GitHub repository rishabhvaibhav/assessment_priceeasy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishabhvaibhav/assessment_priceeasy
- Repository: https://github.com/rishabhvaibhav/assessment_priceeasy

## Overview

#### Repository Scope
This repository contains two independent Python web-scraping projects: a Books to Scrape exporter and a Casey’s store-location scraper. There is no shared package or unified runtime; each project owns its own entry point, scraping target, and CSV outputs.

#### Books to Scrape Script
`Project 1/Book_to_scrape/Scrape_books.py` is a standalone scraper for the Books to Scrape site. It produces the checked-in dataset at `Project 1/Book_to_scrape/books.csv`. Its architecture is a direct fetch/parse/export pipeline rather than a reusable application.

#### Casey’s Modular Application
The primary structured application is under `Project 2/caseys_scraper/Prod_debug`. Its executable entry point is `Project 2/caseys_scraper/Prod_debug/main.py`, which coordinates configuration, request execution, extraction, persistence, and run reporting.

Key runtime inputs are `Project 2/caseys_scraper/Prod_debug/config.json` and `Project 2/caseys_scraper/Prod_debug/requirements.txt`.

#### Configuration and Request Boundary
`Project 2/caseys_scraper/Prod_debug/core/config.py` loads and exposes scraper settings from `Project 2/caseys_scraper/Prod_debug/config.json`. Proxy material is maintained separately in `Project 2/caseys_scraper/Prod_debug/proxies/proxies.txt`.

`Project 2/caseys_scraper/Prod_debug/core/scraper.py` forms the network boundary: it is responsible for obtaining Casey’s source content, likely applying configured request behavior and proxy selection. The external service is Casey’s store-location/web data source; access reliability is an architectural concern because proxy support is explicitly present.

#### Extraction Stage
`Project 2/caseys_scraper/Prod_debug/core/extractor.py` separates parsing and normalization of store records from HTTP retrieval. This boundary lets scraper behavior change independently from the expected store-data schema and makes extracted records suitable for multiple output formats.

#### Storage, State, and Outputs
`Project 2/caseys_scraper/Prod_debug/core/storage.py` persists extracted data and scraper state. The application produces:
- `Project 2/caseys_scraper/Prod_debug/output/caseys_stores_data.csv`
- `Project 2/caseys_scraper/Prod_debug/output/caseys_stores_data.json`
- `Project 2/caseys_scraper/Prod_debug/output/scraped_state.json`
- `Project 2/caseys_scraper/Prod_debug/output/run_summary.json`

The state file indicates the scraper supports progress tracking, resumption, deduplication, or incremental execution. The run summary is a separate operational-reporting artifact rather than primary domain data.

#### Alternate Casey’s Implementation
`Project 2/caseys_scraper/Single Script/caseys_scraper.py` is a separate monolithic implementation of the same Casey’s scraping concern. It writes `Project 2/caseys_scraper/Single Script/caseys_stores_data.csv` and should be treated as an alternative execution path, not a module imported by the modular application.

#### Primary Control Flow
For the modular Casey’s scraper, control flows from `main.py` through configuration loading, optional proxy/input setup, network scraping, record extraction, and storage/reporting. Data crosses three important boundaries: remote Casey’s content into `core/scraper.py`, unstructured response data into normalized records in `core/extractor.py`, and normalized records/state into durable CSV, JSON, and status artifacts via `core/storage.py`.

Main entry points: [`Project 2/caseys_scraper/Prod_debug/core/storage.py`](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/storage.py) (Storage and state), [`Project 2/caseys_scraper/Prod_debug/core/scraper.py`](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/scraper.py) (Request executor), [`Project 2/caseys_scraper/Prod_debug/main.py`](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/main.py) (Run coordinator), [`Project 2/caseys_scraper/Prod_debug/core/config.py`](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/config.py) (Configuration loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_books["Books Exporter"]
  node_books_target{{"Books to Scrape<br/>external source"}}
  node_books_script["Books scraper<br/>Python script<br/>[Scrape_books.py]"]
  node_books_csv["Books dataset<br/>CSV output<br/>[books.csv]"]
end

subgraph group_caseys_modular["Casey's Modular Scraper"]
  node_requirements["Python dependencies<br/>runtime input<br/>[requirements.txt]"]
  node_main["Run coordinator<br/>Python entry point<br/>[main.py]"]
  node_config_json["Scraper settings<br/>JSON configuration<br/>[config.json]"]
  node_config["Configuration loader<br/>Python module<br/>[config.py]"]
  node_proxies["Proxy list<br/>request input<br/>[proxies.txt]"]
  node_caseys_source{{"Casey's store source<br/>external service"}}
  node_scraper["Request executor<br/>Python module<br/>[scraper.py]"]
  node_extractor["Store extractor<br/>Python module<br/>[extractor.py]"]
  node_storage["Storage and state<br/>Python module<br/>[storage.py]"]
  node_caseys_csv["Store CSV<br/>CSV output"]
  node_caseys_json["Store JSON<br/>JSON output"]
  node_scraped_state["Scrape state<br/>JSON state<br/>[scraped_state.json]"]
  node_run_summary["Run summary<br/>JSON report<br/>[run_summary.json]"]
end

subgraph group_caseys_alternate["Casey's Single Script"]
  node_single_script["Casey's single scraper<br/>Python script<br/>[caseys_scraper.py]"]
  node_single_csv["Store CSV<br/>CSV output"]
end

node_books_target -->|"pages"| node_books_script
node_books_script -->|"exports"| node_books_csv
node_requirements -.->|"environment"| node_main
node_main -->|"loads settings"| node_config
node_config_json -->|"read by"| node_config
node_config -->|"request settings"| node_scraper
node_proxies -.->|"proxy material"| node_scraper
node_main -->|"runs"| node_scraper
node_caseys_source -->|"source content"| node_scraper
node_scraper -->|"responses"| node_extractor
node_extractor -->|"normalized records"| node_storage
node_storage -->|"writes"| node_caseys_csv
node_storage -->|"writes"| node_caseys_json
node_storage -->|"updates"| node_scraped_state
node_storage -->|"reports"| node_run_summary
node_caseys_source -->|"source content"| node_single_script
node_single_script -->|"exports"| node_single_csv

click node_books_script "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%201/Book_to_scrape/Scrape_books.py"
click node_books_csv "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%201/Book_to_scrape/books.csv"
click node_requirements "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/requirements.txt"
click node_main "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/main.py"
click node_config_json "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/config.json"
click node_config "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/config.py"
click node_proxies "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/proxies/proxies.txt"
click node_scraper "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/scraper.py"
click node_extractor "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/extractor.py"
click node_storage "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/storage.py"
click node_caseys_csv "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/caseys_stores_data.csv"
click node_caseys_json "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/caseys_stores_data.json"
click node_scraped_state "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/scraped_state.json"
click node_run_summary "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/run_summary.json"
click node_single_script "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Single%20Script/caseys_scraper.py"
click node_single_csv "https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Single%20Script/caseys_stores_data.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_books_target,node_books_script,node_books_csv toneBlue
class node_requirements,node_main,node_config_json,node_config,node_proxies,node_caseys_source,node_scraper,node_extractor,node_storage,node_caseys_csv,node_caseys_json,node_scraped_state,node_run_summary toneAmber
class node_single_script,node_single_csv toneMint
```

## Components

### Books Exporter

Standalone fetch/parse/export script.

- **Books to Scrape**: external source
- [Books scraper](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%201/Book_to_scrape/Scrape_books.py): Python script · `Project 1/Book_to_scrape/Scrape_books.py`
- [Books dataset](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%201/Book_to_scrape/books.csv): CSV output · `Project 1/Book_to_scrape/books.csv`

### Casey's Modular Scraper

Primary structured scraping application.

- [Python dependencies](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/requirements.txt): runtime input · `Project 2/caseys_scraper/Prod_debug/requirements.txt`
- [Run coordinator](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/main.py): Python entry point · Coordinates the modular scraping run. · `Project 2/caseys_scraper/Prod_debug/main.py`
- [Scraper settings](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/config.json): JSON configuration · `Project 2/caseys_scraper/Prod_debug/config.json`
- [Configuration loader](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/config.py): Python module · `Project 2/caseys_scraper/Prod_debug/core/config.py`
- [Proxy list](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/proxies/proxies.txt): request input · `Project 2/caseys_scraper/Prod_debug/proxies/proxies.txt`
- **Casey's store source**: external service
- [Request executor](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/scraper.py): Python module · Network boundary for Casey's source content. · `Project 2/caseys_scraper/Prod_debug/core/scraper.py`
- [Store extractor](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/extractor.py): Python module · Parses and normalizes store records. · `Project 2/caseys_scraper/Prod_debug/core/extractor.py`
- [Storage and state](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/core/storage.py): Python module · Persists records, progress state, and run reporting. · `Project 2/caseys_scraper/Prod_debug/core/storage.py`
- [Store CSV](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/caseys_stores_data.csv): CSV output · `Project 2/caseys_scraper/Prod_debug/output/caseys_stores_data.csv`
- [Store JSON](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/caseys_stores_data.json): JSON output · `Project 2/caseys_scraper/Prod_debug/output/caseys_stores_data.json`
- [Scrape state](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/scraped_state.json): JSON state · Supports incremental or resumable execution. · `Project 2/caseys_scraper/Prod_debug/output/scraped_state.json`
- [Run summary](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Prod_debug/output/run_summary.json): JSON report · `Project 2/caseys_scraper/Prod_debug/output/run_summary.json`

### Casey's Single Script

Alternative monolithic execution path.

- [Casey's single scraper](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Single%20Script/caseys_scraper.py): Python script · Independent alternative to the modular application. · `Project 2/caseys_scraper/Single Script/caseys_scraper.py`
- [Store CSV](https://github.com/rishabhvaibhav/assessment_priceeasy/blob/main/Project%202/caseys_scraper/Single%20Script/caseys_stores_data.csv): CSV output · `Project 2/caseys_scraper/Single Script/caseys_stores_data.csv`

## Connections

- Books to Scrape → Books scraper: pages
- Books scraper → Books dataset: exports
- Python dependencies → Run coordinator: environment
- Run coordinator → Configuration loader: loads settings
- Scraper settings → Configuration loader: read by
- Configuration loader → Request executor: request settings
- Proxy list → Request executor: proxy material
- Run coordinator → Request executor: runs
- Casey's store source → Request executor: source content
- Request executor → Store extractor: responses
- Store extractor → Storage and state: normalized records
- Storage and state → Store CSV: writes
- Storage and state → Store JSON: writes
- Storage and state → Scrape state: updates
- Storage and state → Run summary: reports
- Casey's store source → Casey's single scraper: source content
- Casey's single scraper → Store CSV: exports