# simaremare/transfermarkt-datasets architecture

> Architecture diagram of the GitHub repository simaremare/transfermarkt-datasets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/simaremare/transfermarkt-datasets
- Repository: https://github.com/simaremare/transfermarkt-datasets

## Overview

##### Project Type and Purpose
- This is primarily a data-engineering / ETL pipeline project that acquires raw football (soccer) data from external sources (Transfermarkt API and scraper), stores it in a version‐controlled repository (DVC), transforms it with dbt + DuckDB into curated tables, and then publishes (syncs) the final datasets to various frontends (Kaggle, data.world, and a Streamlit app). Infrastructure (Terraform) and CI/CD (GitHub Actions) glue everything together.

##### File-Structure Analysis
- Top-level folders and files:
- `.github/workflows` – CI/CD pipelines for acquisition, transform, and sync.
- `scripts/acquiring` & `scripts/synching` – Python scripts that pull raw data and push prepared data.
- `data/raw` & `data/prep` – DVC­-​versioned data storage.
- `dbt/` – dbt project: macros, models (raw → curated), tests, profiles.
- `transfermarkt_datasets/` – Python package for a thin API on top of prepared data.
- `infra/` – Terraform definitions for cloud infra (CDN, IAM, etc.).
- `Dockerfile` & `.devcontainer/` – Development container and image for reproducible environment.

##### Additional Insights from README
- Data acquisition is scheduled by GitHub Actions.
- DVC remote storage holds large raw/prep files.
- dbt + DuckDB used locally and in CI to build curated models.
- `make` targets orchestrate local runs of acquire, prepare, and sync.
- Frontends: Streamlit app (Fly.io), Kaggle dataset, data.world dataset.

##### Depth and Detail
- CI workflows by name (e.g., `acquire-transfermarkt-api.yml`, `build.yml`, `sync-kaggle.yml`).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Developer Environment
    subgraph "Developer Environment"
        DE_Docker["Dockerfile"]:::code
        DE_Devcontainer[".devcontainer/devcontainer.json"]:::code
        DE_Makefile["Makefile"]:::code
    end

    %% Source Repository & CI/CD
    subgraph "CI/CD Workflows"
        CI_AcquireAPI["acquire-transfermarkt-api.yml"]:::process
        CI_AcquireScraper["acquire-transfermarkt-scraper.yml"]:::process
        CI_Build["build.yml"]:::process
        CI_BuildContrib["build-contribution.yml"]:::process
        CI_SyncKaggle["sync-kaggle.yml"]:::process
        CI_SyncDataworld["sync-dataworld.yml"]:::process
        CI_SyncStreamlit["sync-streamlit.yml"]:::process
    end

    %% Data Acquisition Layer
    subgraph "Data Acquisition Layer"
        DA_API["transfermarkt-api.py"]:::code
        DA_Scraper["transfermarkt-scraper.py"]:::code
    end

    %% Raw Data Storage
    subgraph "Raw Data Storage"
        RDS_dir["data/raw/"]:::data
        RDS_API_DVC["transfermarkt-api.dvc"]:::data
        RDS_Scraper_DVC["transfermarkt-scraper.dvc"]:::data
    end

    %% Data Transformation Layer
    subgraph "Data Transformation Layer"
        DT_dbtRoot["dbt/dbt_project.yml"]:::code
        DT_profiles["dbt/profiles.yml"]:::code
        DT_macros["dbt/macros/"]:::code
        DT_rawModels["dbt/models/base/"]:::code
        DT_curatedModels["dbt/models/curated/"]:::code
        DT_tests["dbt/tests/"]:::code
        DT_duckdb["dbt/duckdb/"]:::code
    end

    %% Prepared Data Storage
    subgraph "Prepared Data Storage"
        PDS_dvc["data/prep.dvc"]:::data
    end

    %% Python API Layer
    subgraph "Python API Layer"
        API_root["transfermarkt_datasets/"]:::code
        API_core["transfermarkt_datasets/core/"]:::code
        API_assets["transfermarkt_datasets/assets/"]:::code
        API_tests["transfermarkt_datasets/tests/"]:::code
    end

    %% Frontend Publishing
    subgraph "Frontend Publishing"
        FP_Kaggle["scripts/synching/sync-kaggle.py"]:::code
        FP_Dataworld["scripts/synching/sync-dataworld.py"]:::code
        FP_StreamlitConfig[".streamlit/config.toml"]:::code
        FP_FlyConfig["fly.toml"]:::code
    end

    %% Infrastructure as Code
    subgraph "Infrastructure as Code"
        INF_root["infra/main.tf"]:::process
        INF_base["infra/base/base.tf"]:::process
        INF_cdn["infra/cdn/cdn.tf"]:::process
        INF_iam["infra/iam/iam.tf"]:::process
    end

    %% External Services
    subgraph "External Services"
        EXT_TransfermarktAPI["Transfermarkt API"]:::external
        EXT_Website["Transfermarkt Website"]:::external
        EXT_Kaggle["Kaggle"]:::external
        EXT_Dataworld["data.world"]:::external
        EXT_Flyio["Fly.io"]:::external
    end

    %% Developer -> CI
    DE_Docker -->|"push/pr"| CI_AcquireAPI
    DE_Devcontainer -->|"push/pr"| CI_AcquireAPI
    DE_Makefile -->|"local orchestration"| DA_API

    %% CI triggers Acquisition
    CI_AcquireAPI --> DA_API
    CI_AcquireScraper --> DA_Scraper
    DA_API --> RDS_API_DVC
    DA_Scraper --> RDS_Scraper_DVC
    RDS_API_DVC --> RDS_dir
    RDS_Scraper_DVC --> RDS_dir

    %% CI triggers Transformation
    CI_Build --> DT_dbtRoot
    CI_Build --> DT_profiles
    CI_Build --> DT_macros
    CI_Build --> DT_rawModels
    CI_Build --> DT_curatedModels
    CI_Build --> DT_tests
    CI_Build --> DT_duckdb
    RDS_dir --> DT_rawModels
    DT_curatedModels --> PDS_dvc

    %% CI triggers Sync
    CI_SyncKaggle --> FP_Kaggle
    CI_SyncDataworld --> FP_Dataworld
    CI_SyncStreamlit --> FP_StreamlitConfig
    CI_SyncStreamlit --> FP_FlyConfig
    PDS_dvc --> FP_Kaggle
    PDS_dvc --> FP_Dataworld
    PDS_dvc --> FP_StreamlitConfig

    %% Python API usage
    PDS_dvc --> API_root
    API_root --> API_core
    API_root --> API_assets
    API_root --> API_tests

    %% Infra provisioning
    INF_root --> INF_base
    INF_root --> INF_cdn
    INF_root --> INF_iam
    INF_cdn --> EXT_Flyio
    INF_cdn --> EXT_Kaggle
    INF_iam --> EXT_Dataworld

    %% Data sources external
    EXT_TransfermarktAPI --> DA_API
    EXT_Website --> DA_Scraper

    %% Sync to externals
    FP_Kaggle --> EXT_Kaggle
    FP_Dataworld --> EXT_Dataworld
    FP_StreamlitConfig --> EXT_Flyio

    %% Styles
    classDef code fill:#D0E6FF,stroke:#0366d6,color:#0366d6
    classDef data fill:#DFFFE0,stroke:#28a745,color:#22863a
    classDef process fill:#FFE5B4,stroke:#d97706,color:#663c00
    classDef external fill:#E2D5F9,stroke:#6f42c1,color:#5a32a3

    %% Click events
    click DE_Docker "https://github.com/simaremare/transfermarkt-datasets/tree/master/Dockerfile"
    click DE_Devcontainer "https://github.com/simaremare/transfermarkt-datasets/blob/master/.devcontainer/devcontainer.json"
    click DE_Makefile "https://github.com/simaremare/transfermarkt-datasets/tree/master/Makefile"
    click CI_AcquireAPI "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/acquire-transfermarkt-api.yml"
    click CI_AcquireScraper "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/acquire-transfermarkt-scraper.yml"
    click CI_Build "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/build.yml"
    click CI_BuildContrib "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/build-contribution.yml"
    click CI_SyncKaggle "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/sync-kaggle.yml"
    click CI_SyncDataworld "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/sync-dataworld.yml"
    click CI_SyncStreamlit "https://github.com/simaremare/transfermarkt-datasets/blob/master/.github/workflows/sync-streamlit.yml"
    click DA_API "https://github.com/simaremare/transfermarkt-datasets/blob/master/scripts/acquiring/transfermarkt-api.py"
    click DA_Scraper "https://github.com/simaremare/transfermarkt-datasets/blob/master/scripts/acquiring/transfermarkt-scraper.py"
    click RDS_API_DVC "https://github.com/simaremare/transfermarkt-datasets/blob/master/data/raw/transfermarkt-api.dvc"
    click RDS_Scraper_DVC "https://github.com/simaremare/transfermarkt-datasets/blob/master/data/raw/transfermarkt-scraper.dvc"
    click RDS_dir "https://github.com/simaremare/transfermarkt-datasets/tree/master/data/raw/"
    click DT_dbtRoot "https://github.com/simaremare/transfermarkt-datasets/blob/master/dbt/dbt_project.yml"
    click DT_profiles "https://github.com/simaremare/transfermarkt-datasets/blob/master/dbt/profiles.yml"
    click DT_macros "https://github.com/simaremare/transfermarkt-datasets/tree/master/dbt/macros/"
    click DT_rawModels "https://github.com/simaremare/transfermarkt-datasets/tree/master/dbt/models/base/"
    click DT_curatedModels "https://github.com/simaremare/transfermarkt-datasets/tree/master/dbt/models/curated/"
    click DT_tests "https://github.com/simaremare/transfermarkt-datasets/tree/master/dbt/tests/"
    click DT_duckdb "https://github.com/simaremare/transfermarkt-datasets/tree/master/dbt/duckdb/"
    click PDS_dvc "https://github.com/simaremare/transfermarkt-datasets/blob/master/data/prep.dvc"
    click API_root "https://github.com/simaremare/transfermarkt-datasets/tree/master/transfermarkt_datasets/"
    click API_core "https://github.com/simaremare/transfermarkt-datasets/tree/master/transfermarkt_datasets/core/"
    click API_assets "https://github.com/simaremare/transfermarkt-datasets/tree/master/transfermarkt_datasets/assets/"
    click API_tests "https://github.com/simaremare/transfermarkt-datasets/tree/master/transfermarkt_datasets/tests/"
    click FP_Kaggle "https://github.com/simaremare/transfermarkt-datasets/blob/master/scripts/synching/sync-kaggle.py"
    click FP_Dataworld "https://github.com/simaremare/transfermarkt-datasets/blob/master/scripts/synching/sync-dataworld.py"
    click FP_StreamlitConfig "https://github.com/simaremare/transfermarkt-datasets/blob/master/.streamlit/config.toml"
    click FP_FlyConfig "https://github.com/simaremare/transfermarkt-datasets/blob/master/fly.toml"
    click INF_root "https://github.com/simaremare/transfermarkt-datasets/blob/master/infra/main.tf"
    click INF_base "https://github.com/simaremare/transfermarkt-datasets/blob/master/infra/base/base.tf"
    click INF_cdn "https://github.com/simaremare/transfermarkt-datasets/blob/master/infra/cdn/cdn.tf"
    click INF_iam "https://github.com/simaremare/transfermarkt-datasets/blob/master/infra/iam/iam.tf"
```
