# shivam-83/scraping_assignment architecture

> Architecture diagram of the GitHub repository shivam-83/scraping_assignment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shivam-83/scraping_assignment
- Repository: https://github.com/shivam-83/scraping_assignment

## Overview

This repository is a configurable web-scraping ETL pipeline. A command-line run selects the Books and/or Quotes source, crawls paginated pages, and collects records into a shared model; the pipeline then cleans, validates, deduplicates, audits, and exports the results. Checkpoints support crawl recovery, while configuration controls request behavior and output paths. The README documents logging and reports; some wiring to those outputs is not visible in the sampled source, so those links cite the README.

Main entry points: [`main.py`](https://github.com/shivam-83/scraping_assignment/blob/main/main.py) (CLI pipeline), [`scrapers/books_scraper.py`](https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/books_scraper.py) (Books scraper), [`scrapers/quotes_scraper.py`](https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/quotes_scraper.py) (Quotes scraper), [`config.py`](https://github.com/shivam-83/scraping_assignment/blob/main/config.py) (Runtime settings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Run control"]
  node_cli["CLI pipeline<br/>[main.py]"]
  node_settings["Runtime settings<br/>[config.py]"]
  node_settings_file["JSON settings<br/>[config.json]"]
end

subgraph group_extract["Source extraction"]
  node_books["Books scraper<br/>[books_scraper.py]"]
  node_quotes["Quotes scraper<br/>[quotes_scraper.py]"]
  node_http["HTTP and parsing<br/>[base_scraper.py]"]
end

subgraph group_transform["Record processing"]
  node_record["Shared record model<br/>[__init__.py]"]
  node_clean["Record cleaning<br/>[cleaning.py]"]
  node_validate["Record validation<br/>[validation.py]"]
  node_dedupe["Deduplication<br/>[deduplication.py]"]
  node_quality["Quality audit<br/>[data_quality.py]"]
end

subgraph group_persistence["State and outputs"]
  node_checkpoint[("Crawl checkpoints<br/>[checkpoint.py]")]
  node_csv["Consolidated CSV"]
  node_reports["JSON reports"]
  node_logs["Run log"]
end

node_operator(("Operator"))
node_book_source(("Books site"))
node_quote_source(("Quotes site"))

node_operator -->|"runs"| node_cli
node_settings_file -->|"configures"| node_settings
node_cli -->|"resolves options"| node_settings
node_cli -->|"selects and runs"| node_books
node_cli -->|"selects and runs"| node_quotes
node_settings -->|"sets request options"| node_books
node_settings -->|"sets request options"| node_quotes
node_books -->|"inherits HTTP client"| node_http
node_quotes -->|"inherits HTTP client"| node_http
node_http -->|"fetches pages"| node_book_source
node_http -->|"fetches pages"| node_quote_source
node_books -->|"extracts records"| node_record
node_quotes -->|"extracts records"| node_record
node_books -->|"loads and saves progress"| node_checkpoint
node_quotes -->|"loads and saves progress"| node_checkpoint
node_cli -->|"cleans records"| node_clean
node_clean -->|"then validates"| node_validate
node_cli -->|"validates records"| node_validate
node_cli -->|"deduplicates records"| node_dedupe
node_cli -->|"audits dataset"| node_quality
node_quality -->|"checks validity"| node_validate
node_quality -->|"checks uniqueness"| node_dedupe
node_cli -->|"writes dataset"| node_csv
node_cli -->|"writes statistics and quality"| node_reports
node_cli -->|"records events"| node_logs

click node_cli "https://github.com/shivam-83/scraping_assignment/blob/main/main.py"
click node_settings "https://github.com/shivam-83/scraping_assignment/blob/main/config.py"
click node_settings_file "https://github.com/shivam-83/scraping_assignment/blob/main/config.json"
click node_books "https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/books_scraper.py"
click node_quotes "https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/quotes_scraper.py"
click node_http "https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/base_scraper.py"
click node_record "https://github.com/shivam-83/scraping_assignment/blob/main/processing/__init__.py"
click node_clean "https://github.com/shivam-83/scraping_assignment/blob/main/processing/cleaning.py"
click node_validate "https://github.com/shivam-83/scraping_assignment/blob/main/processing/validation.py"
click node_dedupe "https://github.com/shivam-83/scraping_assignment/blob/main/processing/deduplication.py"
click node_quality "https://github.com/shivam-83/scraping_assignment/blob/main/processing/data_quality.py"
click node_checkpoint "https://github.com/shivam-83/scraping_assignment/blob/main/processing/checkpoint.py"
click node_csv "https://github.com/shivam-83/scraping_assignment/tree/main/output"
click node_reports "https://github.com/shivam-83/scraping_assignment/tree/main/output"
click node_logs "https://github.com/shivam-83/scraping_assignment/tree/main/logs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_settings,node_settings_file toneBlue
class node_books,node_quotes,node_http toneAmber
class node_record,node_clean,node_validate,node_dedupe,node_quality toneMint
class node_checkpoint,node_csv,node_reports,node_logs toneRose
class node_operator,node_book_source,node_quote_source toneIndigo
```

## Components

### Run control

- [CLI pipeline](https://github.com/shivam-83/scraping_assignment/blob/main/main.py): `main.py`
- [Runtime settings](https://github.com/shivam-83/scraping_assignment/blob/main/config.py): `config.py`
- [JSON settings](https://github.com/shivam-83/scraping_assignment/blob/main/config.json): `config.json`

### Source extraction

- [Books scraper](https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/books_scraper.py): `scrapers/books_scraper.py`
- [Quotes scraper](https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/quotes_scraper.py): `scrapers/quotes_scraper.py`
- [HTTP and parsing](https://github.com/shivam-83/scraping_assignment/blob/main/scrapers/base_scraper.py): `scrapers/base_scraper.py`

### Record processing

- [Shared record model](https://github.com/shivam-83/scraping_assignment/blob/main/processing/__init__.py): `processing/__init__.py`
- [Record cleaning](https://github.com/shivam-83/scraping_assignment/blob/main/processing/cleaning.py): `processing/cleaning.py`
- [Record validation](https://github.com/shivam-83/scraping_assignment/blob/main/processing/validation.py): `processing/validation.py`
- [Deduplication](https://github.com/shivam-83/scraping_assignment/blob/main/processing/deduplication.py): `processing/deduplication.py`
- [Quality audit](https://github.com/shivam-83/scraping_assignment/blob/main/processing/data_quality.py): `processing/data_quality.py`

### State and outputs

- [Crawl checkpoints](https://github.com/shivam-83/scraping_assignment/blob/main/processing/checkpoint.py): `processing/checkpoint.py`
- [Consolidated CSV](https://github.com/shivam-83/scraping_assignment/tree/main/output): `output`
- [JSON reports](https://github.com/shivam-83/scraping_assignment/tree/main/output): `output`
- [Run log](https://github.com/shivam-83/scraping_assignment/tree/main/logs): `logs`

### Other components

- **Operator**
- **Books site**
- **Quotes site**

## Connections

- Operator → CLI pipeline: runs (evidence: `README.md`)
- JSON settings → Runtime settings: configures (evidence: `README.md`)
- CLI pipeline → Runtime settings: resolves options (evidence: `README.md`)
- CLI pipeline → Books scraper: selects and runs (evidence: `README.md`)
- CLI pipeline → Quotes scraper: selects and runs (evidence: `README.md`)
- Runtime settings → Books scraper: sets request options (evidence: `README.md`)
- Runtime settings → Quotes scraper: sets request options (evidence: `README.md`)
- Books scraper → HTTP and parsing: inherits HTTP client (evidence: `scrapers/books_scraper.py`)
- Quotes scraper → HTTP and parsing: inherits HTTP client (evidence: `scrapers/quotes_scraper.py`)
- HTTP and parsing → Books site: fetches pages (evidence: `README.md`)
- HTTP and parsing → Quotes site: fetches pages (evidence: `README.md`)
- Books scraper → Shared record model: extracts records (evidence: `scrapers/books_scraper.py`)
- Quotes scraper → Shared record model: extracts records (evidence: `scrapers/quotes_scraper.py`)
- Books scraper → Crawl checkpoints: loads and saves progress (evidence: `scrapers/books_scraper.py`)
- Quotes scraper → Crawl checkpoints: loads and saves progress (evidence: `scrapers/quotes_scraper.py`)
- CLI pipeline → Record cleaning: cleans records (evidence: `README.md`)
- Record cleaning → Record validation: then validates (evidence: `README.md`)
- CLI pipeline → Record validation: validates records (evidence: `main.py`)
- CLI pipeline → Deduplication: deduplicates records (evidence: `main.py`)
- CLI pipeline → Quality audit: audits dataset (evidence: `README.md`)
- Quality audit → Record validation: checks validity (evidence: `processing/data_quality.py`)
- Quality audit → Deduplication: checks uniqueness (evidence: `processing/data_quality.py`)
- CLI pipeline → Consolidated CSV: writes dataset (evidence: `README.md`)
- CLI pipeline → JSON reports: writes statistics and quality (evidence: `README.md`)
- CLI pipeline → Run log: records events (evidence: `README.md`)
