How vr3n/data-derby works
Project Type and Purpose
• This is primarily a data‐engineering/ETL pipeline and analytics project for collecting, storing, and analyzing football (soccer) statistics.
• It orchestrates periodic scraping of multiple leagues from the FBRef website into CSVs, loads them into a relational database, version‐controls schema changes, and supports data exploration via Jupyter notebooks.
Key Components
a. External Source
• FBRef.com (external HTTP endpoints providing competition schedules, match pages, player data)
b. Ingestion Layer (Scrapers)
• Python scripts in backend/scraper/ – one per data type (competition schedule, league data, match details, player stats)
• They fetch HTML/JSON, parse content, write CSV files under scraped_data/* or push directly into the DB layer
c. Orchestration & Scheduling
• Apache Airflow (backend/airflow/) running inside Docker via docker-compose.yaml
• DAG definitions (in config/airflow.cfg & Python code) that sequence scraper tasks, SQL‐load tasks, and notebooks
d. Database & Migrations
• Relational DB (e.g., Postgres) provisioned via Docker Compose
• SQLAlchemy models (backend/app/models.py) define schemas
• Alembic (backend/alembic/) handles versioned migrations and schema evolution
e. Data Storage
• Local volume “scraped_data/” containing raw CSVs by league and season
• Persistent DB volume for normalized, queryable tables
f. Data Access & Analysis
• Jupyter notebooks in backend/notebooks/ for ad hoc analysis, pulling from DB or CSVs
• SQL statements in backend/sql_statements/ for standardized queries and inserts
g. Infrastructure & Configuration
• docker-compose.yaml brings up Airflow, the database, and any worker services
• .env.example for environment variables (DB connection, Airflow creds)
• pyproject.toml / poetry for dependency management
• VSCode settings for developer experience
Relationships & Data Flow
1. Airflow Scheduler triggers scraper tasks.
2. Each scraper fetches data from FBRef → outputs CSV to scraped_data/ or pipes rows to backend/app/database.py.
3. Database ingestion tasks run SQL insert statements (backend/sql_statements/) using SQLAlchemy for bulk loads.
4. Alembic ensures schema migrations occur before/after loads.
5. Analytical notebooks query the database for reporting or generate new CSVs.
6. Developers iterate on scraper code, models, and migrations; all orchestrated by Docker Compose.
Architectural Patterns & Principles
• ETL Pipeline: distinct extraction (scrape), transform (parse & normalize models), load (insert into DB) steps.
• Modular Micro‐tasks: separate Python modules for each scrape target, allowing parallel or conditional execution.
• Infrastructure as Code: Docker Compose defines the runtime environment; versioned migrations define the schema.
• Single Responsibility: scraper modules do only scraping; database modules only handle DB sessions; models define data structures.