# parisa-reza/book-web-scraping-pipeline architecture

> Architecture diagram of the GitHub repository parisa-reza/book-web-scraping-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/parisa-reza/book-web-scraping-pipeline
- Repository: https://github.com/parisa-reza/book-web-scraping-pipeline

## Overview

1. **Repository shape**
   - This is a single-purpose **Scrapy** project named `books_web_scraping`.
   - The codebase is centered on one spider and one processing pipeline, with deployment support for both local runs and **Scrapyd**.

2. **Primary crawl source**
   - The crawler targets **`books.toscrape.com`**.
   - The spider discovers book categories, follows category pagination, and samples books from each category, so the crawl is not a flat single-page scrape.

3. **Core crawl component**
   - `books_web_scraping/spiders/book_spider.py` is the main collection entrypoint.
   - It is responsible for navigating the site structure, extracting book fields, and yielding structured items into the Scrapy pipeline.

4. **Data contract**
   - `books_web_scraping/items.py` defines the item shape passed through the system.
   - The output fields are the canonical schema for the rest of the pipeline: `title`, `price`, `availability`, `category`, `product_url`, and `image_url`.

5. **Processing boundary**
   - `books_web_scraping/pipelines.py` is the main transformation and persistence boundary.
   - It cleans or normalizes scraped data and then fans out the same record set to multiple sinks: JSON, CSV, XML, and SQLite.

6. **Storage outputs**
   - The project writes file outputs under `output/` and database output under `data/`.
   - The local/default persistence targets are:
     - `output/books.json`
     - `output/books.csv`
     - `output/books.xml`
     - `data/scraped_books_vault.db`

7. **Database role**
   - SQLite is used as the durable structured store, avoiding any external database server.
   - The table name is `books`, and the table is recreated on each crawl, so each run produces a fresh snapshot rather than an append-only history.

8. **Configuration and runtime behavior**
   - `books_web_scraping/settings.py` holds project-level configuration, including the database path and likely Scrapy pipeline/export settings.
   - `scrapy.cfg` provides the Scrapy project wiring, while `setup.py` packages the module for deployment.

9. **Deployment path**
   - `scrapyd.conf` and the README’s Scrapyd flow show that the spider can be deployed and scheduled over HTTP via **Scrapyd**.
   - In this mode, the crawl is triggered remotely through Scrapyd’s API rather than only by a local `scrapy crawl` command.

10. **Containerized execution**
   - `Dockerfile` packages the project so Scrapyd can run inside a container.
   - Host-mounted volumes separate runtime artifacts from the image: `docker-output/` for exported files and `docker-data/` for the SQLite database.

11. **Main data flow**
   - The architecture is linear:
     - `books.toscrape.com` → `book_spider` → `BooksWebScrapingItem` → `BooksWebScrapingPipeline` → JSON/CSV/XML + SQLite
   - The spider produces normalized item objects, and the pipeline is the single point where all persistence happens.

12. **Operational boundaries**
   - There are two execution modes with the same crawl logic:
     - **Local Scrapy execution** for direct crawling and output generation
     - **Scrapyd-managed execution** for API-triggered runs and containerized hosting
   - The crawl logic stays in the Scrapy project; deployment only changes how the spider is launched.

13. **Important architectural constraint**
   - This project is tightly coupled to the structure of `books.toscrape.com`.
   - The site-specific selectors and category traversal logic are the main place where changes would be needed if the source site changed.

Main entry points: [`books_web_scraping/pipelines.py`](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/pipelines.py) (Pipeline), [`books_web_scraping/spiders/book_spider.py`](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/spiders/book_spider.py) (Book spider), [`scrapy.cfg`](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/scrapy.cfg) (scrapy.cfg), [`books_web_scraping/items.py`](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/items.py) (Book item)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Scrapy app"]
  node_n2["Book spider<br/>[book_spider.py]"]
  node_n3["Book item<br/>item schema<br/>[items.py]"]
  node_n4["Pipeline<br/>transform+sinks<br/>[pipelines.py]"]
  node_n5["Settings<br/>runtime config<br/>[settings.py]"]
  node_n6{{"Spider package"}}
end

subgraph group_g2["Persistence"]
  node_n7["JSON export<br/>file sink"]
  node_n8["CSV export<br/>file sink"]
  node_n9["XML export<br/>file sink"]
  node_n10[("SQLite vault<br/>database")]
  node_n11[("books table<br/>sqlite table")]
end

subgraph group_g3["Deployment"]
  node_n12(("Scrapyd<br/>job runner"))
  node_n13["scrapyd.conf<br/>deploy config<br/>[scrapyd.conf]"]
  node_n14{{"Docker image<br/>container image"}}
  node_n15["setup.py<br/>package metadata<br/>[setup.py]"]
  node_n16["scrapy.cfg<br/>project wiring<br/>[scrapy.cfg]"]
  node_n17(("Local run<br/>execution mode"))
  node_n18(("Remote run<br/>execution mode"))
end

node_n1(("books.toscrape.com<br/>target site"))

node_n1 -->|"crawl target"| node_n2
node_n2 -->|"yields"| node_n3
node_n3 -->|"processed by"| node_n4
node_n5 -.->|"configures"| node_n4
node_n6 -->|"contains"| node_n2
node_n4 -->|"exports"| node_n7
node_n4 -->|"exports"| node_n8
node_n4 -->|"exports"| node_n9
node_n4 -->|"stores"| node_n10
node_n10 -->|"contains"| node_n11
node_n2 -.->|"runs in"| node_n17
node_n2 -.->|"runs in"| node_n18
node_n13 -.->|"configures"| node_n12
node_n14 -.->|"hosts"| node_n12
node_n15 -.->|"installs"| node_n12
node_n16 -.->|"wires"| node_n17
node_n16 -.->|"wires"| node_n18
node_n18 -->|"triggered via"| node_n12
node_n17 -->|"feeds"| node_n4
node_n18 -->|"feeds"| node_n4

click node_n2 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/spiders/book_spider.py"
click node_n3 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/items.py"
click node_n4 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/pipelines.py"
click node_n5 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/settings.py"
click node_n6 "https://github.com/parisa-reza/book-web-scraping-pipeline/tree/main/books_web_scraping/spiders"
click node_n13 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/scrapyd.conf"
click node_n14 "https://github.com/parisa-reza/book-web-scraping-pipeline/tree/main/Dockerfile"
click node_n15 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/setup.py"
click node_n16 "https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/scrapy.cfg"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n2,node_n3,node_n4,node_n5,node_n6 toneBlue
class node_n7,node_n8,node_n9,node_n10,node_n11 toneAmber
class node_n12,node_n13,node_n14,node_n15,node_n16,node_n17,node_n18 toneMint
class node_n1 toneNeutral
```

## Components

### Scrapy app

Core crawl, item, and pipeline code inside the Python package.

- [Book spider](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/spiders/book_spider.py): spider · Main crawler that traverses categories, paginates listings, and extracts book records. · `books_web_scraping/spiders/book_spider.py`
- [Book item](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/items.py): item schema · Canonical record shape passed through the Scrapy pipeline. · `books_web_scraping/items.py`
- [Pipeline](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/pipelines.py): transform+sinks · Normalizes scraped data and routes records to all persistence targets. · `books_web_scraping/pipelines.py`
- [Settings](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/books_web_scraping/settings.py): runtime config · Project-level Scrapy configuration for paths, pipelines, and deployment behavior. · `books_web_scraping/settings.py`
- [Spider package](https://github.com/parisa-reza/book-web-scraping-pipeline/tree/main/books_web_scraping/spiders): package · Namespace containing spider definitions. · `books_web_scraping/spiders`

### Persistence

Logical storage targets produced by the pipeline.

- **JSON export**: file sink · Logical JSON output written by the pipeline.
- **CSV export**: file sink · Logical CSV output written by the pipeline.
- **XML export**: file sink · Logical XML output written by the pipeline.
- **SQLite vault**: database · Durable local SQLite store for crawl snapshots.
- **books table**: sqlite table · Fresh table recreated on each crawl run.

### Deployment

Packaging and Scrapyd runtime support.

- **Scrapyd**: job runner · HTTP-triggered runtime for remote spider execution.
- [scrapyd.conf](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/scrapyd.conf): deploy config · Configuration for running the project under Scrapyd. · `scrapyd.conf`
- [Docker image](https://github.com/parisa-reza/book-web-scraping-pipeline/tree/main/Dockerfile): container image · Container package for Scrapyd and runtime artifacts. · `Dockerfile`
- [setup.py](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/setup.py): package metadata · Packaging metadata used for installation and deployment. · `setup.py`
- [scrapy.cfg](https://github.com/parisa-reza/book-web-scraping-pipeline/blob/main/scrapy.cfg): project wiring · Scrapy project entry wiring for local commands and project loading. · `scrapy.cfg`
- **Local run**: execution mode · Direct Scrapy execution for local crawling and output generation.
- **Remote run**: execution mode · Scrapyd-managed execution triggered through the API.

### Other components

- **books.toscrape.com**: target site · Site-specific crawl target with category, pagination, and product pages.

## Connections

- books.toscrape.com → Book spider: crawl target · The spider is tailored to the structure of books.toscrape.com.
- Book spider → Book item: yields · The spider emits structured book items.
- Book item → Pipeline: processed by · Pipeline consumes the canonical item schema.
- Settings → Pipeline: configures · Settings provide pipeline and runtime configuration.
- Spider package → Book spider: contains · The spider lives in the spiders package.
- Pipeline → JSON export: exports · Pipeline writes JSON output.
- Pipeline → CSV export: exports · Pipeline writes CSV output.
- Pipeline → XML export: exports · Pipeline writes XML output.
- Pipeline → SQLite vault: stores · Pipeline persists crawl data to SQLite.
- SQLite vault → books table: contains · SQLite database stores the books table snapshot.
- Book spider → Local run: runs in · Local Scrapy runs launch the spider directly.
- Book spider → Remote run: runs in · Remote runs launch the same spider under Scrapyd.
- scrapyd.conf → Scrapyd: configures · Scrapyd uses its dedicated configuration.
- Docker image → Scrapyd: hosts · The container image packages the Scrapyd runtime.
- setup.py → Scrapyd: installs · Packaging metadata enables deployment as an installable project.
- scrapy.cfg → Local run: wires · Scrapy project wiring supports direct local execution.
- scrapy.cfg → Remote run: wires · Scrapy project wiring also supports Scrapyd execution.
- Remote run → Scrapyd: triggered via · Remote crawls are started over Scrapyd's API.
- Local run → Pipeline: feeds · Local runs send scraped items into the pipeline.
- Remote run → Pipeline: feeds · Remote runs use the same item-to-pipeline flow.