How federicolaco/f1-data-pipeline works
This is an F1 season-data pipeline: Jolpica data is fetched into raw JSON, loaded into PostgreSQL staging, promoted into analytics tables, and queried for standings; quality checks are also part of the documented workflow. It can run manually or as an Airflow DAG. The sampled Python shows extraction, raw-file handling, staging/analytics loading, configuration, and DAG task calls. SQL capabilities and database details are represented from the README and file tree; their implementations were not sampled.
Main entry points: postgres_loader.py Postgres loader · jolpica_client.py Season extractor · f1_pipeline.py Airflow DAG · 02_analytics.sql Analytics tables
Components 11
Ingestion
- Pipeline configuration
src/config.py - Season extractor
src/extract/jolpica_client.py - Raw JSON files
data/raw
Warehouse
- Postgres loader
src/load/postgres_loader.py - Staging tables
sql/01_staging.sql - Analytics tables
sql/02_analytics.sql
Orchestration
- Airflow DAG
dags/f1_pipeline.py
Quality and queries
- Data quality checks
sql/03_checks.sql - Driver standings query
sql/04_driver_standings.sql
Other components
- Pipeline operator
- Jolpica API
Connections 14
- Pipeline operator → Season extractor: runs manually
README.md - Pipeline operator → Airflow DAG: triggers
README.md - Airflow DAG → Season extractor: calls extract
dags/f1_pipeline.py - Airflow DAG → Postgres loader: calls load tasks
dags/f1_pipeline.py - Season extractor → Pipeline configuration: reads settings
src/extract/jolpica_client.py - Postgres loader → Pipeline configuration: reads settings
src/load/postgres_loader.py - Season extractor → Jolpica API: fetches season data
src/extract/jolpica_client.py - Season extractor → Raw JSON files: writes JSON
src/extract/jolpica_client.py - Postgres loader → Raw JSON files: reads JSON
src/load/postgres_loader.py - Postgres loader → Staging tables: loads records
src/load/postgres_loader.py - Postgres loader → Analytics tables: promotes staging data
src/load/postgres_loader.py - Data quality checks → Staging tables: checks data
- Data quality checks → Analytics tables: checks data
- Driver standings query → Analytics tables: queries standings
README.md