How probberechts/soccerdata works
Project type and purpose
• SoccerData is a Python library of web‐scraping “scraper” classes that fetch soccer data from several external websites (Club Elo, ESPN, FBref, etc.) and return Pandas DataFrames with a unified schema.
• It is essentially an open‐source data‐acquisition tool, not a monolithic application or microservice.
File structure
• Top level:
• soccerdata/ — the core package
• docs/ — Sphinx documentation, notebooks, examples
• tests/ — pytest test suite, canned config and JSON fixtures
• config/build files: pyproject.toml, Makefile, noxfile.py, .github/workflows
• Within soccerdata/:
• _common.py, _config.py — shared base classes, configuration and caching logic
• clubelo.py, espn.py, fbref.py, … understat.py, whoscored.py — one module per scraper source
• Configuration & caching: uses local files under tests/appdata/config; likely a cache directory under the user’s home or temp.
README for additional insights
• Each scraper is a class (e.g. FBref) instantiated with league & season, exposing methods like read_schedule(), read_team_season_stats(), etc.
• Data is downloaded on demand and cached locally.
• API is synchronous, returns Pandas DataFrames.
• No centralized DB or message bus—external websites are the only “services.”
Details
• Caching strategy (e.g. “Checks FS cache before HTTP fetch”).
• Shared base class in _common.py as the abstraction point.
• Which config values live in _config.py vs. user‐exposed parameters.
• Don’t over‐modularize—group all 8+ external sites behind a single “External Services” node with a sub‐list.