How ks415/bvp-scraper-python works
Project Type and Purpose
• This is an open-source Python library (not a full-stack app) whose sole purpose is to scrape boat race data (programs, previews, odds, results, stadium info) from the official Boatrace website.
• It exposes a simple facade (Scraper class) that users call with dates/race IDs, under the hood delegating work to a set of specialized scrapers.
File Structure Analysis
• Top-level:
• bvp_scraper/ ← main package
– __init__.py ← exports public API (Scraper)
– base_scraper.py ← abstract base class defining common scraping workflow
– interfaces.py ← type hints & data models for scraped items
– scraper_core.py ← orchestrates HTTP fetch, retries, parsing, and dispatch to concrete scrapers
– scraper.py ← public‐facing facade with static methods (scrape_programs, scrape_odds, etc.)
– scrapers/ ← concrete scraper modules (program, preview, odds, result, stadium)
• Other directories:
• examples/ & example_usage.py ← usage demos
• debug/ ← ad-hoc scripts for selector testing & debugging
• tests/ ← pytest suites (unit & integration)
• Configuration: pyproject.toml for dependencies (requests, BeautifulSoup4, tenacity, etc.), Makefile/uv config for build/test.
README Insights
• Clarifies architectural lineage: ported from PHP, maintains same design patterns but with Python idioms.
• Highlights third-party libraries: requests for HTTP, BeautifulSoup4/lxml for parsing, tenacity for retry logic, dateutil for date handling.
• Emphasizes API methods as facade calls, type hints for safety, and testing via pytest.
Essential Details to Capture
• The split between public API (Scraper) and internal workflow (scraper_core, base_scraper, scrapers).
• Retry & error-handling loop around HTTP requests.
• Parsing step distinct from data extraction step.
• Clear depiction of how new scrapers plug into Core.
• Return data flow back to the user.