# ks415/bvp-scraper-python architecture

> Architecture diagram of the GitHub repository ks415/bvp-scraper-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ks415/bvp-scraper-python
- Repository: https://github.com/ks415/bvp-scraper-python

## Overview

##### Project Type and Purpose
- This is an open-source Python library (not a full-stack app) whose sole purpose is to scrape boat race data (programs, previews, odds, results, stadium info) from the official Boatrace website.
- It exposes a simple facade (`Scraper` class) that users call with dates/race IDs, under the hood delegating work to a set of specialized scrapers.

##### File Structure Analysis
- Top-level:
- bvp_scraper/         ← main package
– __init__.py        ← exports public API (Scraper)
– base_scraper.py    ← abstract base class defining common scraping workflow
– interfaces.py      ← type hints & data models for scraped items
– scraper_core.py    ← orchestrates HTTP fetch, retries, parsing, and dispatch to concrete scrapers
– scraper.py         ← public‐facing facade with static methods (scrape_programs, scrape_odds, etc.)
– scrapers/          ← concrete scraper modules (program, preview, odds, result, stadium)
- Other directories:
- examples/ & example_usage.py  ← usage demos
- debug/                         ← ad-hoc scripts for selector testing & debugging
- tests/                         ← pytest suites (unit & integration)
- Configuration: pyproject.toml for dependencies (requests, BeautifulSoup4, tenacity, etc.), Makefile/uv config for build/test.

##### README Insights
- Clarifies architectural lineage: ported from PHP, maintains same design patterns but with Python idioms.
- Highlights third-party libraries: requests for HTTP, BeautifulSoup4/lxml for parsing, tenacity for retry logic, dateutil for date handling.
- Emphasizes API methods as facade calls, type hints for safety, and testing via pytest.

##### Essential Details to Capture
- The split between public API (Scraper) and internal workflow (scraper_core, base_scraper, scrapers).
- Retry & error-handling loop around HTTP requests.
- Parsing step distinct from data extraction step.
- Clear depiction of how new scrapers plug into Core.
- Return data flow back to the user.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User
    User["User/Client Code"]:::external

    %% Core Package
    subgraph "bvp_scraper"
        direction TB
        ScraperFacade["Scraper (Facade)"]:::internal
        CoreOrchestrator["ScraperCore (Orchestrator)"]:::internal
        BaseScraper["BaseScraper (Template Method)"]:::internal
        Interfaces["Interfaces & Data Models"]:::model

        subgraph "Concrete Scrapers"
            direction TB
            ProgramScraper["ProgramScraper"]:::internal
            PreviewScraper["PreviewScraper"]:::internal
            OddsScraper["OddsScraper"]:::internal
            ResultScraper["ResultScraper"]:::internal
            StadiumScraper["StadiumScraper"]:::internal
        end
    end

    %% Third-Party Libraries
    Requests["Requests"]:::thirdparty
    Tenacity["Tenacity (Retry)"]:::thirdparty
    BS4["BeautifulSoup4"]:::thirdparty
    LXML["lxml"]:::thirdparty

    %% External Service
    ExternalWebsite["Boatrace Official Website"]:::external_service

    %% Examples, Debug, Tests
    subgraph examples["Examples"]
        ExampleUsage["example_usage.py"]:::external
        ExamplesDir["examples/"]:::external
    end
    subgraph debug["Debug"]
        DebugDir["debug/"]:::external
    end
    subgraph tests["Tests"]
        TestsDir["tests/"]:::external
    end

    %% Data Flow
    User -->|"calls scrape_*"| ScraperFacade
    ScraperFacade -->|"delegates to"| CoreOrchestrator
    CoreOrchestrator -->|"uses"| BaseScraper
    CoreOrchestrator -->|"defines models"| Interfaces
    CoreOrchestrator -->|"injects"| Requests
    CoreOrchestrator -->|"injects"| Tenacity
    Requests & Tenacity -->|"GET requests"| ExternalWebsite
    ExternalWebsite -->|"HTML response"| CoreOrchestrator
    CoreOrchestrator -->|"parses HTML"| BS4
    BS4 -->|"uses parser backend"| LXML
    CoreOrchestrator -->|"dispatches DOM to"| ProgramScraper
    CoreOrchestrator -->|"dispatches DOM to"| PreviewScraper
    CoreOrchestrator -->|"dispatches DOM to"| OddsScraper
    CoreOrchestrator -->|"dispatches DOM to"| ResultScraper
    CoreOrchestrator -->|"dispatches DOM to"| StadiumScraper
    ProgramScraper -->|"extracts data into"| Interfaces
    PreviewScraper -->|"extracts data into"| Interfaces
    OddsScraper -->|"extracts data into"| Interfaces
    ResultScraper -->|"extracts data into"| Interfaces
    StadiumScraper -->|"extracts data into"| Interfaces
    Interfaces -->|"models returned to"| CoreOrchestrator
    CoreOrchestrator -->|"returns data to"| ScraperFacade
    ScraperFacade -->|"returns to user"| User

    %% Click Events
    click ScraperFacade "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scraper.py"
    click CoreOrchestrator "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scraper_core.py"
    click BaseScraper "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/base_scraper.py"
    click Interfaces "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/interfaces.py"
    click ProgramScraper "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scrapers/program_scraper.py"
    click PreviewScraper "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scrapers/preview_scraper.py"
    click OddsScraper "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scrapers/odds_scraper.py"
    click ResultScraper "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scrapers/result_scraper.py"
    click StadiumScraper "https://github.com/ks415/bvp-scraper-python/blob/main/bvp_scraper/scrapers/stadium_scraper.py"
    click ExampleUsage "https://github.com/ks415/bvp-scraper-python/blob/main/example_usage.py"
    click ExamplesDir "https://github.com/ks415/bvp-scraper-python/tree/main/examples/"
    click DebugDir "https://github.com/ks415/bvp-scraper-python/tree/main/debug/"
    click TestsDir "https://github.com/ks415/bvp-scraper-python/tree/main/tests/"

    %% Styles
    classDef internal fill:#cfe2f3,stroke:#000,color:#000
    classDef thirdparty fill:#d5ead1,stroke:#000,color:#000
    classDef external fill:#f4cccc,stroke:#000,color:#000
    classDef external_service fill:#f4cccc,stroke:#000,color:#000,shape:cylinder
    classDef model fill:#fff2cc,stroke:#000,color:#000
```
