How simaremare/transfermarkt-datasets works
Project Type and Purpose
• This is primarily a data-engineering / ETL pipeline project that acquires raw football (soccer) data from external sources (Transfermarkt API and scraper), stores it in a version‐controlled repository (DVC), transforms it with dbt + DuckDB into curated tables, and then publishes (syncs) the final datasets to various frontends (Kaggle, data.world, and a Streamlit app). Infrastructure (Terraform) and CI/CD (GitHub Actions) glue everything together.
File-Structure Analysis
• Top-level folders and files:
• .github/workflows – CI/CD pipelines for acquisition, transform, and sync.
• scripts/acquiring & scripts/synching – Python scripts that pull raw data and push prepared data.
• data/raw & data/prep – DVC-versioned data storage.
• dbt/ – dbt project: macros, models (raw → curated), tests, profiles.
• transfermarkt_datasets/ – Python package for a thin API on top of prepared data.
• infra/ – Terraform definitions for cloud infra (CDN, IAM, etc.).
• Dockerfile & .devcontainer/ – Development container and image for reproducible environment.
Additional Insights from README
• Data acquisition is scheduled by GitHub Actions.
• DVC remote storage holds large raw/prep files.
• dbt + DuckDB used locally and in CI to build curated models.
• make targets orchestrate local runs of acquire, prepare, and sync.
• Frontends: Streamlit app (Fly.io), Kaggle dataset, data.world dataset.
Depth and Detail
• CI workflows by name (e.g., acquire-transfermarkt-api.yml, build.yml, sync-kaggle.yml).