How parisa-reza/book-web-scraping-pipeline works
1. Repository shape
- This is a single-purpose Scrapy project named books_web_scraping.
- The codebase is centered on one spider and one processing pipeline, with deployment support for both local runs and Scrapyd.
2. Primary crawl source
- The crawler targets books.toscrape.com.
- The spider discovers book categories, follows category pagination, and samples books from each category, so the crawl is not a flat single-page scrape.
3. Core crawl component
- books_web_scraping/spiders/book_spider.py is the main collection entrypoint.
- It is responsible for navigating the site structure, extracting book fields, and yielding structured items into the Scrapy pipeline.
4. Data contract
- books_web_scraping/items.py defines the item shape passed through the system.
- The output fields are the canonical schema for the rest of the pipeline: title, price, availability, category, product_url, and image_url.
5. Processing boundary
- books_web_scraping/pipelines.py is the main transformation and persistence boundary.
- It cleans or normalizes scraped data and then fans out the same record set to multiple sinks: JSON, CSV, XML, and SQLite.
6. Storage outputs
- The project writes file outputs under output/ and database output under data/.
- The local/default persistence targets are:
- output/books.json
- output/books.csv
- output/books.xml
- data/scraped_books_vault.db
7. Database role
- SQLite is used as the durable structured store, avoiding any external database server.
- The table name is books, and the table is recreated on each crawl, so each run produces a fresh snapshot rather than an append-only history.
8. Configuration and runtime behavior
- books_web_scraping/settings.py holds project-level configuration, including the database path and likely Scrapy pipeline/export settings.
- scrapy.cfg provides the Scrapy project wiring, while setup.py packages the module for deployment.
9. Deployment path
- scrapyd.conf and the README’s Scrapyd flow show that the spider can be deployed and scheduled over HTTP via Scrapyd.
- In this mode, the crawl is triggered remotely through Scrapyd’s API rather than only by a local scrapy crawl command.
10. Containerized execution
- Dockerfile packages the project so Scrapyd can run inside a container.
- Host-mounted volumes separate runtime artifacts from the image: docker-output/ for exported files and docker-data/ for the SQLite database.
11. Main data flow
- The architecture is linear:
- books.toscrape.com → book_spider → BooksWebScrapingItem → BooksWebScrapingPipeline → JSON/CSV/XML + SQLite
- The spider produces normalized item objects, and the pipeline is the single point where all persistence happens.
12. Operational boundaries
- There are two execution modes with the same crawl logic:
- Local Scrapy execution for direct crawling and output generation
- Scrapyd-managed execution for API-triggered runs and containerized hosting
- The crawl logic stays in the Scrapy project; deployment only changes how the spider is launched.
13. Important architectural constraint
- This project is tightly coupled to the structure of books.toscrape.com.
- The site-specific selectors and category traversal logic are the main place where changes would be needed if the source site changed.
Main entry points: pipelines.py Pipeline · book_spider.py Book spider · scrapy.cfg scrapy.cfg · items.py Book item
Components 18
Scrapy app
Core crawl, item, and pipeline code inside the Python package.
- Book spider · Main crawler that traverses categories, paginates listings, and extracts book records.
books_web_scraping/spiders/book_spider.py - Book item · Canonical record shape passed through the Scrapy pipeline.
books_web_scraping/items.py - Pipeline · Normalizes scraped data and routes records to all persistence targets.
books_web_scraping/pipelines.py - Settings · Project-level Scrapy configuration for paths, pipelines, and deployment behavior.
books_web_scraping/settings.py - Spider package · Namespace containing spider definitions.
books_web_scraping/spiders
Persistence
Logical storage targets produced by the pipeline.
- JSON export · Logical JSON output written by the pipeline.
- CSV export · Logical CSV output written by the pipeline.
- XML export · Logical XML output written by the pipeline.
- SQLite vault · Durable local SQLite store for crawl snapshots.
- books table · Fresh table recreated on each crawl run.
Deployment
Packaging and Scrapyd runtime support.
- Scrapyd · HTTP-triggered runtime for remote spider execution.
- scrapyd.conf · Configuration for running the project under Scrapyd.
scrapyd.conf - Docker image · Container package for Scrapyd and runtime artifacts.
Dockerfile - setup.py · Packaging metadata used for installation and deployment.
setup.py - scrapy.cfg · Scrapy project entry wiring for local commands and project loading.
scrapy.cfg - Local run · Direct Scrapy execution for local crawling and output generation.
- Remote run · Scrapyd-managed execution triggered through the API.
Other components
- books.toscrape.com · Site-specific crawl target with category, pagination, and product pages.
Connections 20
- books.toscrape.com → Book spider: crawl target · The spider is tailored to the structure of books.toscrape.com.
- Book spider → Book item: yields · The spider emits structured book items.
- Book item → Pipeline: processed by · Pipeline consumes the canonical item schema.
- Settings → Pipeline: configures · Settings provide pipeline and runtime configuration.
- Spider package → Book spider: contains · The spider lives in the spiders package.
- Pipeline → JSON export: exports · Pipeline writes JSON output.
- Pipeline → CSV export: exports · Pipeline writes CSV output.
- Pipeline → XML export: exports · Pipeline writes XML output.
- Pipeline → SQLite vault: stores · Pipeline persists crawl data to SQLite.
- SQLite vault → books table: contains · SQLite database stores the books table snapshot.
- Book spider → Local run: runs in · Local Scrapy runs launch the spider directly.
- Book spider → Remote run: runs in · Remote runs launch the same spider under Scrapyd.
- scrapyd.conf → Scrapyd: configures · Scrapyd uses its dedicated configuration.
- Docker image → Scrapyd: hosts · The container image packages the Scrapyd runtime.
- setup.py → Scrapyd: installs · Packaging metadata enables deployment as an installable project.
- scrapy.cfg → Local run: wires · Scrapy project wiring supports direct local execution.
- scrapy.cfg → Remote run: wires · Scrapy project wiring also supports Scrapyd execution.
- Remote run → Scrapyd: triggered via · Remote crawls are started over Scrapyd's API.
- Local run → Pipeline: feeds · Local runs send scraped items into the pipeline.
- Remote run → Pipeline: feeds · Remote runs use the same item-to-pipeline flow.