# benjaminbeni1/book-catalogue-scraper architecture

> Architecture diagram of the GitHub repository benjaminbeni1/book-catalogue-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/benjaminbeni1/book-catalogue-scraper
- Repository: https://github.com/benjaminbeni1/book-catalogue-scraper

## Overview

This is a small, single-file catalogue scraper. Running `scraper.py` fetches catalogue pages with retries and delays, extracts and normalizes book fields into an in-memory collection, then writes the collection to SQLite and CSV. The README also documents pagination and skipped failed pages. The runtime is monolithic, so the diagram separates its stages by responsibility rather than implying separate modules; all code-stage paths point to the inspected `scraper.py`.

Main entry points: [`scraper.py`](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py) (Scraper entry point)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Scraping runtime"]
  node_scraper["Scraper entry point<br/>[scraper.py]"]
  node_fetch["Page fetching<br/>[scraper.py]"]
  node_parse["HTML extraction<br/>[scraper.py]"]
  node_records["Book records<br/>[scraper.py]"]
end

subgraph group_outputs["Data outputs"]
  node_sqlite[("SQLite database<br/>[scraper.py]")]
  node_csv["CSV export<br/>[scraper.py]"]
end

node_user(("Terminal user"))
node_catalogue["Book catalogue"]

node_user -->|"runs"| node_scraper
node_scraper -->|"fetches pages"| node_fetch
node_fetch -->|"requests pages"| node_catalogue
node_catalogue -->|"returns HTML"| node_fetch
node_scraper -->|"parses HTML"| node_parse
node_parse -->|"extracts fields"| node_records
node_scraper -->|"writes records"| node_sqlite
node_scraper -->|"exports records"| node_csv

click node_scraper "https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py"
click node_fetch "https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py"
click node_parse "https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py"
click node_records "https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py"
click node_sqlite "https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py"
click node_csv "https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper,node_fetch,node_parse,node_records,node_user toneBlue
class node_sqlite,node_csv toneAmber
class node_catalogue toneIndigo
```

## Components

### Scraping runtime

- [Scraper entry point](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py): `scraper.py`
- [Page fetching](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py): `scraper.py`
- [HTML extraction](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py): `scraper.py`
- [Book records](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py): `scraper.py`

### Data outputs

- [SQLite database](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py): `scraper.py`
- [CSV export](https://github.com/benjaminbeni1/book-catalogue-scraper/blob/main/scraper.py): `scraper.py`

### Other components

- **Terminal user**
- **Book catalogue**

## Connections

- Terminal user → Scraper entry point: runs
- Scraper entry point → Page fetching: fetches pages
- Page fetching → Book catalogue: requests pages
- Book catalogue → Page fetching: returns HTML
- Scraper entry point → HTML extraction: parses HTML
- HTML extraction → Book records: extracts fields
- Scraper entry point → SQLite database: writes records
- Scraper entry point → CSV export: exports records