# vr3n/data-derby architecture

> Architecture diagram of the GitHub repository vr3n/data-derby, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vr3n/data-derby
- Repository: https://github.com/vr3n/data-derby

## Overview

##### Project Type and Purpose
- This is primarily a data‐engineering/ETL pipeline and analytics project for collecting, storing, and analyzing football (soccer) statistics.
- It orchestrates periodic scraping of multiple leagues from the FBRef website into CSVs, loads them into a relational database, version‐controls schema changes, and supports data exploration via Jupyter notebooks.

##### Key Components
a. External Source
- FBRef.com (external HTTP endpoints providing competition schedules, match pages, player data)
b. Ingestion Layer (Scrapers)
- Python scripts in backend/scraper/ – one per data type (competition schedule, league data, match details, player stats)
- They fetch HTML/JSON, parse content, write CSV files under scraped_data/* or push directly into the DB layer
c. Orchestration & Scheduling
- Apache Airflow (backend/airflow/) running inside Docker via docker-compose.yaml
- DAG definitions (in config/airflow.cfg & Python code) that sequence scraper tasks, SQL‐load tasks, and notebooks
d. Database & Migrations
- Relational DB (e.g., Postgres) provisioned via Docker Compose
- SQLAlchemy models (backend/app/models.py) define schemas
- Alembic (backend/alembic/) handles versioned migrations and schema evolution
e. Data Storage
- Local volume “scraped_data/” containing raw CSVs by league and season
- Persistent DB volume for normalized, queryable tables
f. Data Access & Analysis
- Jupyter notebooks in backend/notebooks/ for ad hoc analysis, pulling from DB or CSVs
- SQL statements in backend/sql_statements/ for standardized queries and inserts
g. Infrastructure & Configuration
- docker-compose.yaml brings up Airflow, the database, and any worker services
- .env.example for environment variables (DB connection, Airflow creds)
- pyproject.toml / poetry for dependency management
- VSCode settings for developer experience

##### Relationships & Data Flow
1. Airflow Scheduler triggers scraper tasks.
2. Each scraper fetches data from FBRef → outputs CSV to scraped_data/ or pipes rows to backend/app/database.py.
3. Database ingestion tasks run SQL insert statements (backend/sql_statements/) using SQLAlchemy for bulk loads.
4. Alembic ensures schema migrations occur before/after loads.
5. Analytical notebooks query the database for reporting or generate new CSVs.
6. Developers iterate on scraper code, models, and migrations; all orchestrated by Docker Compose.

##### Architectural Patterns & Principles
- ETL Pipeline: distinct extraction (scrape), transform (parse & normalize models), load (insert into DB) steps.
- Modular Micro‐tasks: separate Python modules for each scrape target, allowing parallel or conditional execution.
- Infrastructure as Code: Docker Compose defines the runtime environment; versioned migrations define the schema.
- Single Responsibility: scraper modules do only scraping; database modules only handle DB sessions; models define data structures.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Source
    FBRef(("FBRef Service")):::external

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        CompScraper["Competition Scraper\n(Python)"]:::ingestion
        LegScraper["League Data Scraper\n(Python)"]:::ingestion
        MatchScraper["Match Details Scraper\n(Python)"]:::ingestion
        PlayerScraper["Player Stats Scraper\n(Python)"]:::ingestion
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        AirflowScheduler["Airflow Scheduler\n(DAGs)"]:::orchestration
        DockerCompose["Docker Compose\n(Services)"]:::orchestration
        AirflowConfig["Airflow Config\n(airflow.cfg)"]:::orchestration
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        RawCSV["Raw CSV Volume\n(scraped_data/)"]:::storage
        Alembic["Alembic Migration Engine"]:::storage
        Database[(PostgreSQL DB)]:::storage
    end

    %% Analysis Layer
    subgraph "Analysis Layer"
        direction TB
        Notebooks["Jupyter Notebooks\n(.ipynb)"]:::analysis
        SQLTemplates["SQL Templates\n(insert_*.sql)"]:::analysis
    end

    %% Developer Layer
    subgraph "Developer Workstation"
        direction TB
        IDE["VSCode Settings"]:::developer
        EnvExample[".env.example"]:::developer
        Poetry["pyproject.toml"]:::developer
        Models["DB Models\n(models.py)"]:::developer
        DBEngine["DB Session & Engine\n(database.py)"]:::developer
        AppMain["App Entry Point\n(main.py)"]:::developer
        AppConfig["App Config\n(config.py)"]:::developer
        AlembicConfig["alembic.ini & env.py"]:::developer
    end

    %% Data Flows
    FBRef -->|scrapes| CompScraper
    FBRef -->|scrapes| LegScraper
    FBRef -->|scrapes| MatchScraper
    FBRef -->|scrapes| PlayerScraper

    CompScraper -->|writes| RawCSV
    LegScraper -->|writes| RawCSV
    MatchScraper -->|writes| RawCSV
    PlayerScraper -->|writes| RawCSV

    AirflowScheduler -. triggers .-> CompScraper
    AirflowScheduler -. triggers .-> LegScraper
    AirflowScheduler -. triggers .-> MatchScraper
    AirflowScheduler -. triggers .-> PlayerScraper
    DockerCompose -->|orchestrates| AirflowScheduler
    DockerCompose -->|orchestrates| Database

    RawCSV -->|bulk load| AppMain
    AppMain -->|uses| Models
    AppMain -->|uses| DBEngine
    AppMain -->|runs migrations via| Alembic

    Alembic -->|migrates| Database

    Database -->|read/write| SQLTemplates
    Notebooks -->|query| Database
    Notebooks -->|read CSV| RawCSV

    IDE -->|edits| CompScraper
    IDE -->|edits| Models
    IDE -->|edits| AlembicConfig

    %% Click Events
    click CompScraper "https://github.com/vr3n/data-derby/blob/master/backend/scraper/fbref_competition_schedule_scrape.py"
    click LegScraper "https://github.com/vr3n/data-derby/blob/master/backend/scraper/fbref_league_data_scrape.py"
    click MatchScraper "https://github.com/vr3n/data-derby/blob/master/backend/scraper/fbref_match_scraper.py"
    click PlayerScraper "https://github.com/vr3n/data-derby/blob/master/backend/scraper/fbref_players_data_scrape.py"
    click AirflowConfig "https://github.com/vr3n/data-derby/blob/master/backend/airflow/config/airflow.cfg"
    click DockerCompose "https://github.com/vr3n/data-derby/blob/master/docker-compose.yaml"
    click Models "https://github.com/vr3n/data-derby/blob/master/backend/app/models.py"
    click DBEngine "https://github.com/vr3n/data-derby/blob/master/backend/app/database.py"
    click AppMain "https://github.com/vr3n/data-derby/blob/master/backend/app/main.py"
    click AppConfig "https://github.com/vr3n/data-derby/blob/master/backend/app/config.py"
    click AlembicConfig "https://github.com/vr3n/data-derby/blob/master/backend/alembic.ini"
    click AlembicConfig "https://github.com/vr3n/data-derby/blob/master/backend/alembic/env.py"
    click AlembicConfig "https://github.com/vr3n/data-derby/blob/master/backend/alembic/script.py.mako"
    click RawCSV "https://github.com/vr3n/data-derby/tree/master/scraped_data/"
    click SQLTemplates "https://github.com/vr3n/data-derby/blob/master/backend/sql_statements/insert_competitions.sql"
    click SQLTemplates "https://github.com/vr3n/data-derby/blob/master/backend/sql_statements/insert_countries.sql"
    click SQLTemplates "https://github.com/vr3n/data-derby/blob/master/backend/sql_statements/insert_seasons.sql"
    click SQLTemplates "https://github.com/vr3n/data-derby/blob/master/backend/sql_statements/query_playerstats_data.sql"
    click Notebooks "https://github.com/vr3n/data-derby/blob/master/backend/notebooks/Untitled.ipynb"
    click Poetry "https://github.com/vr3n/data-derby/blob/master/pyproject.toml"
    click EnvExample "https://github.com/vr3n/data-derby/blob/master/.env.example"
    click IDE "https://github.com/vr3n/data-derby/tree/master/.vscode/"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#1f78b4,color:#1f1f1f
    classDef orchestration fill:#d5f5e3,stroke:#27ae60,color:#1f1f1f
    classDef storage fill:#f9e79f,stroke:#d4ac0d,color:#1f1f1f
    classDef analysis fill:#fadbd8,stroke:#c0392b,color:#1f1f1f
    classDef developer fill:#f5cba7,stroke:#ba4a00,color:#1f1f1f
    classDef external fill:#d6eaf8,stroke:#2980b9,color:#1f1f1f
```
