# parasu-bw/scraping-model architecture

> Architecture diagram of the GitHub repository parasu-bw/scraping-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/parasu-bw/scraping-model
- Repository: https://github.com/parasu-bw/scraping-model

## Overview

Scraping-Model collects product-page data through two paths: it can fetch JavaScript-rendered pages into HTML and batch-parse them, or scrape URLs directly in a Playwright session. Both paths extract product name, price, and description and write timestamped JSON results. The batch path separates acquisition, HTML parsing, and product extraction. The README gives no further capability detail; the direct scraper and batch processor are evidenced by source excerpts.

Main entry points: [`Model1/main_processor.py`](https://github.com/parasu-bw/scraping-model/blob/main/Model1/main_processor.py) (Batch processor), [`Model1/fetcher.py`](https://github.com/parasu-bw/scraping-model/blob/main/Model1/fetcher.py) (HTML fetcher), [`Model1/playwright_scraper.py`](https://github.com/parasu-bw/scraping-model/blob/main/Model1/playwright_scraper.py) (Direct scraper), [`Model1/parser.py`](https://github.com/parasu-bw/scraping-model/blob/main/Model1/parser.py) (HTML parser)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs"]
  node_url_csv["URL list<br/>[prod_url.csv]"]
end

subgraph group_acquisition["Page Acquisition"]
  node_html_fetcher["HTML fetcher<br/>[fetcher.py]"]
  node_html_pages["Saved HTML"]
  node_direct_scraper["Direct scraper"]
end

subgraph group_processing["Extraction"]
  node_html_parser["HTML parser<br/>[parser.py]"]
  node_smart_extractor["Product extractor<br/>[smart_extractor.py]"]
  node_batch_processor["Batch processor<br/>[main_processor.py]"]
end

subgraph group_results["Results"]
  node_batch_results["Batch JSON"]
  node_direct_results["Direct JSON"]
end

node_operator(("User"))
node_product_sites(("Product websites"))

node_operator -->|"provides CSV path"| node_html_fetcher
node_html_fetcher -->|"reads URLs"| node_url_csv
node_html_fetcher -->|"fetches pages"| node_product_sites
node_html_fetcher -->|"saves HTML"| node_html_pages
node_operator -->|"provides CSV path"| node_direct_scraper
node_direct_scraper -->|"reads URLs"| node_url_csv
node_direct_scraper -->|"visits pages"| node_product_sites
node_direct_scraper -->|"writes JSON"| node_direct_results
node_operator -->|"runs processing"| node_batch_processor
node_batch_processor -->|"reads files"| node_html_pages
node_batch_processor -->|"calls parser"| node_html_parser
node_html_parser -->|"returns soup"| node_batch_processor
node_batch_processor -->|"calls extractor"| node_smart_extractor
node_smart_extractor -->|"returns product data"| node_batch_processor
node_batch_processor -->|"writes JSON"| node_batch_results

click node_url_csv "https://github.com/parasu-bw/scraping-model/blob/main/Model1/prod_url.csv"
click node_html_fetcher "https://github.com/parasu-bw/scraping-model/blob/main/Model1/fetcher.py"
click node_html_pages "https://github.com/parasu-bw/scraping-model/tree/main/Model1/html_files"
click node_direct_scraper "https://github.com/parasu-bw/scraping-model/blob/main/Model1/playwright_scraper.py"
click node_html_parser "https://github.com/parasu-bw/scraping-model/blob/main/Model1/parser.py"
click node_smart_extractor "https://github.com/parasu-bw/scraping-model/blob/main/Model1/smart_extractor.py"
click node_batch_processor "https://github.com/parasu-bw/scraping-model/blob/main/Model1/main_processor.py"
click node_batch_results "https://github.com/parasu-bw/scraping-model/blob/main/Model1/scraped_product_data_2025-09-11_15-38-50.json"
click node_direct_results "https://github.com/parasu-bw/scraping-model/blob/main/Model1/playwright_scraped_data_2025-09-11_14-54-32.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_url_csv,node_operator toneBlue
class node_html_fetcher,node_html_pages,node_direct_scraper toneAmber
class node_html_parser,node_smart_extractor,node_batch_processor toneMint
class node_batch_results,node_direct_results toneRose
class node_product_sites toneIndigo
```

## Components

### Inputs

- [URL list](https://github.com/parasu-bw/scraping-model/blob/main/Model1/prod_url.csv): `Model1/prod_url.csv`

### Page Acquisition

- [HTML fetcher](https://github.com/parasu-bw/scraping-model/blob/main/Model1/fetcher.py): `Model1/fetcher.py`
- [Saved HTML](https://github.com/parasu-bw/scraping-model/tree/main/Model1/html_files): `Model1/html_files`
- [Direct scraper](https://github.com/parasu-bw/scraping-model/blob/main/Model1/playwright_scraper.py): `Model1/playwright_scraper.py`

### Extraction

- [HTML parser](https://github.com/parasu-bw/scraping-model/blob/main/Model1/parser.py): `Model1/parser.py`
- [Product extractor](https://github.com/parasu-bw/scraping-model/blob/main/Model1/smart_extractor.py): `Model1/smart_extractor.py`
- [Batch processor](https://github.com/parasu-bw/scraping-model/blob/main/Model1/main_processor.py): `Model1/main_processor.py`

### Results

- [Batch JSON](https://github.com/parasu-bw/scraping-model/blob/main/Model1/scraped_product_data_2025-09-11_15-38-50.json): `Model1/scraped_product_data_2025-09-11_15-38-50.json`
- [Direct JSON](https://github.com/parasu-bw/scraping-model/blob/main/Model1/playwright_scraped_data_2025-09-11_14-54-32.json): `Model1/playwright_scraped_data_2025-09-11_14-54-32.json`

### Other components

- **User**
- **Product websites**

## Connections

- User → HTML fetcher: provides CSV path (evidence: `Model1/fetcher.py`)
- HTML fetcher → URL list: reads URLs (evidence: `Model1/fetcher.py`)
- HTML fetcher → Product websites: fetches pages (evidence: `Model1/fetcher.py`)
- HTML fetcher → Saved HTML: saves HTML (evidence: `Model1/fetcher.py`)
- User → Direct scraper: provides CSV path (evidence: `Model1/playwright_scraper.py`)
- Direct scraper → URL list: reads URLs (evidence: `Model1/playwright_scraper.py`)
- Direct scraper → Product websites: visits pages (evidence: `Model1/playwright_scraper.py`)
- Direct scraper → Direct JSON: writes JSON (evidence: `Model1/playwright_scraper.py`)
- User → Batch processor: runs processing (evidence: `Model1/main_processor.py`)
- Batch processor → Saved HTML: reads files (evidence: `Model1/main_processor.py`)
- Batch processor → HTML parser: calls parser (evidence: `Model1/main_processor.py`)
- HTML parser → Batch processor: returns soup (evidence: `Model1/main_processor.py`)
- Batch processor → Product extractor: calls extractor (evidence: `Model1/main_processor.py`)
- Product extractor → Batch processor: returns product data (evidence: `Model1/main_processor.py`)
- Batch processor → Batch JSON: writes JSON (evidence: `Model1/main_processor.py`)
