How shivam-83/scraping_assignment works
This repository is a configurable web-scraping ETL pipeline. A command-line run selects the Books and/or Quotes source, crawls paginated pages, and collects records into a shared model; the pipeline then cleans, validates, deduplicates, audits, and exports the results. Checkpoints support crawl recovery, while configuration controls request behavior and output paths. The README documents logging and reports; some wiring to those outputs is not visible in the sampled source, so those links cite the README.
Main entry points: main.py CLI pipeline · books_scraper.py Books scraper · quotes_scraper.py Quotes scraper · config.py Runtime settings
Components 18
Run control
- CLI pipeline
main.py - Runtime settings
config.py - JSON settings
config.json
Source extraction
- Books scraper
scrapers/books_scraper.py - Quotes scraper
scrapers/quotes_scraper.py - HTTP and parsing
scrapers/base_scraper.py
Record processing
- Shared record model
processing/__init__.py - Record cleaning
processing/cleaning.py - Record validation
processing/validation.py - Deduplication
processing/deduplication.py - Quality audit
processing/data_quality.py
State and outputs
- Crawl checkpoints
processing/checkpoint.py - Consolidated CSV
output - JSON reports
output - Run log
logs
Other components
- Operator
- Books site
- Quotes site
Connections 25
- Operator → CLI pipeline: runs
README.md - JSON settings → Runtime settings: configures
README.md - CLI pipeline → Runtime settings: resolves options
README.md - CLI pipeline → Books scraper: selects and runs
README.md - CLI pipeline → Quotes scraper: selects and runs
README.md - Runtime settings → Books scraper: sets request options
README.md - Runtime settings → Quotes scraper: sets request options
README.md - Books scraper → HTTP and parsing: inherits HTTP client
scrapers/books_scraper.py - Quotes scraper → HTTP and parsing: inherits HTTP client
scrapers/quotes_scraper.py - HTTP and parsing → Books site: fetches pages
README.md - HTTP and parsing → Quotes site: fetches pages
README.md - Books scraper → Shared record model: extracts records
scrapers/books_scraper.py - Quotes scraper → Shared record model: extracts records
scrapers/quotes_scraper.py - Books scraper → Crawl checkpoints: loads and saves progress
scrapers/books_scraper.py - Quotes scraper → Crawl checkpoints: loads and saves progress
scrapers/quotes_scraper.py - CLI pipeline → Record cleaning: cleans records
README.md - Record cleaning → Record validation: then validates
README.md - CLI pipeline → Record validation: validates records
main.py - CLI pipeline → Deduplication: deduplicates records
main.py - CLI pipeline → Quality audit: audits dataset
README.md - Quality audit → Record validation: checks validity
processing/data_quality.py - Quality audit → Deduplication: checks uniqueness
processing/data_quality.py - CLI pipeline → Consolidated CSV: writes dataset
README.md - CLI pipeline → JSON reports: writes statistics and quality
README.md - CLI pipeline → Run log: records events
README.md