douglasbc/scraping-understat-dataset

1. This repository is a data acquisition and packaging project, not an application runtime. Its main purpose is to scrape Understat and publish cleaned CSV datasets for football/soccer analytics.

2. The external system boundary is Understat.com. All primary data originates there and is pulled into the repo by Python scraping scripts, then stored locally as versioned CSV outputs.

3. The scraping subsystem lives under scraping/. It is split into:

- season_scraper.py for season-level collection,

- match_scraper.py for match-level collection,

- update.py as the orchestration entry point for refreshing the dataset.

4. The update workflow appears to be incremental rather than a one-off export. The README says to place scraping and datasets side by side and run update.py, which suggests the script reads existing CSVs, fetches new Understat data, and appends or refreshes files in place.

5. League-specific configuration is isolated in scraping/teams_set/. Each text file contains team lists for one competition:

- EPL

- La Liga

- Bundesliga

- Serie A

- Ligue 1

- RFPL

These files likely provide name normalization and league scoping for the scraper.

6. The repository’s core data products are under datasets/, organized first by league and then by season. For each league there are paired outputs:

- players_<league>_<season>.csv

- shots_<league>_<season>.csv

Seasons run from 14-15 through 21-22.

7. There is also a repository-wide aggregate file at datasets/shots_dataset.csv, which likely combines all leagues and seasons into one shots table for downstream analysis. The per-league directories provide the modular source-of-truth slices.

8. The data model is split into two domains:

- shot-level event data, which is the main Understat xG dataset,

- player-season summary data, which captures season stats for players involved in those matches.

The separate file families indicate different row grains and should be diagrammed as distinct dataset outputs.

9. Documentation is handled separately in documentation/ through PDF data dictionaries for shots and players. This is the schema/semantics boundary for analysts who consume the CSVs.

10. requirements.txt defines the Python runtime dependencies for the scraping pipeline. No web service, database, or queueing system is present in the repo; the architecture is a local Python ETL-style workflow that writes flat files.

11. scraping/empty_url_update.txt suggests there is some operational bookkeeping for missing or unresolved Understat URLs during updates. That implies the scraper includes failure handling or partial-refresh behavior.

12. For an architecture diagram, the clearest flow is: Understat source site -> Python scraping/update scripts -> league/team configuration -> per-league/per-season CSV outputs -> combined datasets and documentation for consumers.