# douglasbc/scraping-understat-dataset architecture

> Architecture diagram of the GitHub repository douglasbc/scraping-understat-dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/douglasbc/scraping-understat-dataset
- Repository: https://github.com/douglasbc/scraping-understat-dataset

## Overview

1. This repository is a data acquisition and packaging project, not an application runtime. Its main purpose is to scrape Understat and publish cleaned CSV datasets for football/soccer analytics.

2. The external system boundary is Understat.com. All primary data originates there and is pulled into the repo by Python scraping scripts, then stored locally as versioned CSV outputs.

3. The scraping subsystem lives under `scraping/`. It is split into:
   - `season_scraper.py` for season-level collection,
   - `match_scraper.py` for match-level collection,
   - `update.py` as the orchestration entry point for refreshing the dataset.

4. The update workflow appears to be incremental rather than a one-off export. The README says to place `scraping` and `datasets` side by side and run `update.py`, which suggests the script reads existing CSVs, fetches new Understat data, and appends or refreshes files in place.

5. League-specific configuration is isolated in `scraping/teams_set/`. Each text file contains team lists for one competition:
   - EPL
   - La Liga
   - Bundesliga
   - Serie A
   - Ligue 1
   - RFPL  
   These files likely provide name normalization and league scoping for the scraper.

6. The repository’s core data products are under `datasets/`, organized first by league and then by season. For each league there are paired outputs:
   - `players_<league>_<season>.csv`
   - `shots_<league>_<season>.csv`  
   Seasons run from `14-15` through `21-22`.

7. There is also a repository-wide aggregate file at `datasets/shots_dataset.csv`, which likely combines all leagues and seasons into one shots table for downstream analysis. The per-league directories provide the modular source-of-truth slices.

8. The data model is split into two domains:
   - shot-level event data, which is the main Understat xG dataset,
   - player-season summary data, which captures season stats for players involved in those matches.  
   The separate file families indicate different row grains and should be diagrammed as distinct dataset outputs.

9. Documentation is handled separately in `documentation/` through PDF data dictionaries for shots and players. This is the schema/semantics boundary for analysts who consume the CSVs.

10. `requirements.txt` defines the Python runtime dependencies for the scraping pipeline. No web service, database, or queueing system is present in the repo; the architecture is a local Python ETL-style workflow that writes flat files.

11. `scraping/empty_url_update.txt` suggests there is some operational bookkeeping for missing or unresolved Understat URLs during updates. That implies the scraper includes failure handling or partial-refresh behavior.

12. For an architecture diagram, the clearest flow is: Understat source site -> Python scraping/update scripts -> league/team configuration -> per-league/per-season CSV outputs -> combined datasets and documentation for consumers.

Main entry points: [`scraping/match_scraper.py`](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/match_scraper.py) (Match scraper), [`scraping/season_scraper.py`](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/season_scraper.py) (Season scraper), [`scraping/update.py`](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/update.py) (Update script), [`documentation/shots_data_dictionary.pdf`](https://github.com/douglasbc/scraping-understat-dataset/blob/main/documentation/shots_data_dictionary.pdf) (Shots dictionary)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_source["External source"]
  node_understat(("Understat<br/>external source"))
end

subgraph group_pipeline["Scraping pipeline"]
  node_update["Update script<br/>orchestrator<br/>[update.py]"]
  node_season_scraper["Season scraper<br/>collector<br/>[season_scraper.py]"]
  node_match_scraper["Match scraper<br/>collector<br/>[match_scraper.py]"]
  node_teams_set["Team lists<br/>league config"]
  node_empty_url["URL log<br/>ops log"]
  node_requirements["Dependencies<br/>python deps<br/>[requirements.txt]"]
end

subgraph group_outputs["Data products"]
  node_shots_dataset[("Shots dataset<br/>aggregate csv<br/>[shots_dataset.csv]")]
  node_epl_data[("EPL outputs<br/>league csvs")]
  node_laliga_data[("La Liga outputs<br/>league csvs")]
  node_bundesliga_data[("Bundesliga outputs<br/>league csvs")]
  node_seriea_data[("Serie A outputs<br/>league csvs")]
  node_ligue1_data[("Ligue 1 outputs<br/>league csvs")]
  node_rfpl_data[("RFPL outputs<br/>league csvs")]
  node_shots_docs["Shots dictionary<br/>documentation"]
  node_players_docs["Players dictionary<br/>documentation"]
end

node_understat -->|"source data"| node_update
node_update -->|"invokes"| node_season_scraper
node_update -->|"invokes"| node_match_scraper
node_update -->|"uses"| node_teams_set
node_update -.->|"records issues"| node_empty_url
node_update -.->|"depends on"| node_requirements
node_season_scraper -->|"scrapes"| node_understat
node_match_scraper -->|"scrapes"| node_understat
node_season_scraper -->|"normalizes"| node_teams_set
node_match_scraper -->|"normalizes"| node_teams_set
node_season_scraper -->|"writes"| node_epl_data
node_season_scraper -->|"writes"| node_laliga_data
node_season_scraper -->|"writes"| node_bundesliga_data
node_season_scraper -->|"writes"| node_seriea_data
node_season_scraper -->|"writes"| node_ligue1_data
node_season_scraper -->|"writes"| node_rfpl_data
node_match_scraper -->|"writes"| node_epl_data
node_match_scraper -->|"writes"| node_laliga_data
node_match_scraper -->|"writes"| node_bundesliga_data
node_match_scraper -->|"writes"| node_seriea_data
node_match_scraper -->|"writes"| node_ligue1_data
node_match_scraper -->|"writes"| node_rfpl_data
node_match_scraper -->|"consolidates"| node_shots_dataset
node_epl_data -.->|"described by"| node_shots_docs
node_epl_data -.->|"described by"| node_players_docs
node_laliga_data -.->|"described by"| node_shots_docs
node_laliga_data -.->|"described by"| node_players_docs
node_bundesliga_data -.->|"described by"| node_shots_docs
node_bundesliga_data -.->|"described by"| node_players_docs
node_seriea_data -.->|"described by"| node_shots_docs
node_seriea_data -.->|"described by"| node_players_docs
node_ligue1_data -.->|"described by"| node_shots_docs
node_ligue1_data -.->|"described by"| node_players_docs
node_rfpl_data -.->|"described by"| node_shots_docs
node_rfpl_data -.->|"described by"| node_players_docs

click node_update "https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/update.py"
click node_season_scraper "https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/season_scraper.py"
click node_match_scraper "https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/match_scraper.py"
click node_teams_set "https://github.com/douglasbc/scraping-understat-dataset/tree/main/scraping/teams_set"
click node_empty_url "https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/empty_url_update.txt"
click node_requirements "https://github.com/douglasbc/scraping-understat-dataset/blob/main/requirements.txt"
click node_shots_dataset "https://github.com/douglasbc/scraping-understat-dataset/blob/main/datasets/shots_dataset.csv"
click node_epl_data "https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/epl"
click node_laliga_data "https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/la_liga"
click node_bundesliga_data "https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/bundesliga"
click node_seriea_data "https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/serie_a"
click node_ligue1_data "https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/ligue_1"
click node_rfpl_data "https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/rfpl"
click node_shots_docs "https://github.com/douglasbc/scraping-understat-dataset/blob/main/documentation/shots_data_dictionary.pdf"
click node_players_docs "https://github.com/douglasbc/scraping-understat-dataset/blob/main/documentation/players_data_dictionary.pdf"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_understat toneBlue
class node_update,node_season_scraper,node_match_scraper,node_teams_set,node_empty_url,node_requirements toneAmber
class node_shots_dataset,node_epl_data,node_laliga_data,node_bundesliga_data,node_seriea_data,node_ligue1_data,node_rfpl_data,node_shots_docs,node_players_docs toneMint
```

## Components

### External source

Primary data origin outside the repository.

- **Understat**: external source · Source site for football match and player data.

### Scraping pipeline

Python ETL flow that refreshes and normalizes Understat data.

- [Update script](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/update.py): orchestrator · Entry point that refreshes datasets incrementally. · `scraping/update.py`
- [Season scraper](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/season_scraper.py): collector · Collects season-level player data from Understat. · `scraping/season_scraper.py`
- [Match scraper](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/match_scraper.py): collector · Collects match-level shot data from Understat. · `scraping/match_scraper.py`
- [Team lists](https://github.com/douglasbc/scraping-understat-dataset/tree/main/scraping/teams_set): league config · League-specific team lists used for scoping and name normalization. · `scraping/teams_set`
- [URL log](https://github.com/douglasbc/scraping-understat-dataset/blob/main/scraping/empty_url_update.txt): ops log · Bookkeeping for unresolved or missing Understat URLs during updates. · `scraping/empty_url_update.txt`
- [Dependencies](https://github.com/douglasbc/scraping-understat-dataset/blob/main/requirements.txt): python deps · Python packages required to run the scraping pipeline. · `requirements.txt`

### Data products

Versioned CSV datasets and analyst-facing documentation.

- [Shots dataset](https://github.com/douglasbc/scraping-understat-dataset/blob/main/datasets/shots_dataset.csv): aggregate csv · Repository-wide combined shots table across leagues and seasons. · `datasets/shots_dataset.csv`
- [EPL outputs](https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/epl): league csvs · Per-season player and shot CSV outputs for the EPL. · `datasets/epl`
- [La Liga outputs](https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/la_liga): league csvs · Per-season player and shot CSV outputs for La Liga. · `datasets/la_liga`
- [Bundesliga outputs](https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/bundesliga): league csvs · Per-season player and shot CSV outputs for Bundesliga. · `datasets/bundesliga`
- [Serie A outputs](https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/serie_a): league csvs · Per-season player and shot CSV outputs for Serie A. · `datasets/serie_a`
- [Ligue 1 outputs](https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/ligue_1): league csvs · Per-season player and shot CSV outputs for Ligue 1. · `datasets/ligue_1`
- [RFPL outputs](https://github.com/douglasbc/scraping-understat-dataset/tree/main/datasets/rfpl): league csvs · Per-season player and shot CSV outputs for RFPL. · `datasets/rfpl`
- [Shots dictionary](https://github.com/douglasbc/scraping-understat-dataset/blob/main/documentation/shots_data_dictionary.pdf): documentation · PDF data dictionary for shot-level CSV fields and semantics. · `documentation/shots_data_dictionary.pdf`
- [Players dictionary](https://github.com/douglasbc/scraping-understat-dataset/blob/main/documentation/players_data_dictionary.pdf): documentation · PDF data dictionary for player-season CSV fields and semantics. · `documentation/players_data_dictionary.pdf`

## Connections

- Understat → Update script: source data · The update workflow pulls fresh data from Understat.
- Update script → Season scraper: invokes · Refresh orchestration delegates season-level collection.
- Update script → Match scraper: invokes · Refresh orchestration delegates match-level collection.
- Update script → Team lists: uses · League team lists guide normalization and scoping.
- Update script → URL log: records issues · Missing or unresolved URLs are tracked during updates.
- Update script → Dependencies: depends on · Python runtime dependencies support the scraping pipeline.
- Season scraper → Understat: scrapes · Season-level data is collected from the external site.
- Match scraper → Understat: scrapes · Match-level shot data is collected from Understat.
- Season scraper → Team lists: normalizes · Team lists help map league-specific names.
- Match scraper → Team lists: normalizes · Team lists help map league-specific names.
- Season scraper → EPL outputs: writes · League-season player CSV outputs are produced for EPL.
- Season scraper → La Liga outputs: writes · League-season player CSV outputs are produced for La Liga.
- Season scraper → Bundesliga outputs: writes · League-season player CSV outputs are produced for Bundesliga.
- Season scraper → Serie A outputs: writes · League-season player CSV outputs are produced for Serie A.
- Season scraper → Ligue 1 outputs: writes · League-season player CSV outputs are produced for Ligue 1.
- Season scraper → RFPL outputs: writes · League-season player CSV outputs are produced for RFPL.
- Match scraper → EPL outputs: writes · League-season shot CSV outputs are produced for EPL.
- Match scraper → La Liga outputs: writes · League-season shot CSV outputs are produced for La Liga.
- Match scraper → Bundesliga outputs: writes · League-season shot CSV outputs are produced for Bundesliga.
- Match scraper → Serie A outputs: writes · League-season shot CSV outputs are produced for Serie A.
- Match scraper → Ligue 1 outputs: writes · League-season shot CSV outputs are produced for Ligue 1.
- Match scraper → RFPL outputs: writes · League-season shot CSV outputs are produced for RFPL.
- Match scraper → Shots dataset: consolidates · Per-league shot slices feed the combined shots dataset.
- EPL outputs → Shots dictionary: described by · Data dictionary documents shot CSV semantics.
- EPL outputs → Players dictionary: described by · Data dictionary documents player CSV semantics.
- La Liga outputs → Shots dictionary: described by · Data dictionary documents shot CSV semantics.
- La Liga outputs → Players dictionary: described by · Data dictionary documents player CSV semantics.
- Bundesliga outputs → Shots dictionary: described by · Data dictionary documents shot CSV semantics.
- Bundesliga outputs → Players dictionary: described by · Data dictionary documents player CSV semantics.
- Serie A outputs → Shots dictionary: described by · Data dictionary documents shot CSV semantics.
- Serie A outputs → Players dictionary: described by · Data dictionary documents player CSV semantics.
- Ligue 1 outputs → Shots dictionary: described by · Data dictionary documents shot CSV semantics.
- Ligue 1 outputs → Players dictionary: described by · Data dictionary documents player CSV semantics.
- RFPL outputs → Shots dictionary: described by · Data dictionary documents shot CSV semantics.
- RFPL outputs → Players dictionary: described by · Data dictionary documents player CSV semantics.