# federicolaco/f1-data-pipeline architecture

> Architecture diagram of the GitHub repository federicolaco/f1-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/federicolaco/f1-data-pipeline
- Repository: https://github.com/federicolaco/f1-data-pipeline

## Overview

This is an F1 season-data pipeline: Jolpica data is fetched into raw JSON, loaded into PostgreSQL staging, promoted into analytics tables, and queried for standings; quality checks are also part of the documented workflow. It can run manually or as an Airflow DAG. The sampled Python shows extraction, raw-file handling, staging/analytics loading, configuration, and DAG task calls. SQL capabilities and database details are represented from the README and file tree; their implementations were not sampled.

Main entry points: [`src/load/postgres_loader.py`](https://github.com/federicolaco/f1-data-pipeline/blob/main/src/load/postgres_loader.py) (Postgres loader), [`src/extract/jolpica_client.py`](https://github.com/federicolaco/f1-data-pipeline/blob/main/src/extract/jolpica_client.py) (Season extractor), [`dags/f1_pipeline.py`](https://github.com/federicolaco/f1-data-pipeline/blob/main/dags/f1_pipeline.py) (Airflow DAG), [`sql/02_analytics.sql`](https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/02_analytics.sql) (Analytics tables)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_config["Pipeline configuration<br/>[config.py]"]
  node_extract["Season extractor<br/>[jolpica_client.py]"]
  node_raw["Raw JSON files"]
end

subgraph group_warehouse["Warehouse"]
  node_loader["Postgres loader<br/>[postgres_loader.py]"]
  node_staging[("Staging tables<br/>[01_staging.sql]")]
  node_analytics_tables[("Analytics tables<br/>[02_analytics.sql]")]
end

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow DAG<br/>[f1_pipeline.py]"]
end

subgraph group_analytics["Quality and queries"]
  node_quality["Data quality checks<br/>[03_checks.sql]"]
  node_standings["Driver standings query"]
end

node_operator(("Pipeline operator"))
node_jolpica{{"Jolpica API"}}

node_operator -->|"runs manually"| node_extract
node_operator -->|"triggers"| node_airflow
node_airflow -->|"calls extract"| node_extract
node_airflow -->|"calls load tasks"| node_loader
node_extract -->|"reads settings"| node_config
node_loader -->|"reads settings"| node_config
node_extract -->|"fetches season data"| node_jolpica
node_extract -->|"writes JSON"| node_raw
node_loader -->|"reads JSON"| node_raw
node_loader -->|"loads records"| node_staging
node_loader -->|"promotes staging data"| node_analytics_tables
node_quality -.->|"checks data"| node_staging
node_quality -.->|"checks data"| node_analytics_tables
node_standings -->|"queries standings"| node_analytics_tables

click node_config "https://github.com/federicolaco/f1-data-pipeline/blob/main/src/config.py"
click node_extract "https://github.com/federicolaco/f1-data-pipeline/blob/main/src/extract/jolpica_client.py"
click node_raw "https://github.com/federicolaco/f1-data-pipeline/tree/main/data/raw"
click node_loader "https://github.com/federicolaco/f1-data-pipeline/blob/main/src/load/postgres_loader.py"
click node_staging "https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/01_staging.sql"
click node_analytics_tables "https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/02_analytics.sql"
click node_airflow "https://github.com/federicolaco/f1-data-pipeline/blob/main/dags/f1_pipeline.py"
click node_quality "https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/03_checks.sql"
click node_standings "https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/04_driver_standings.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_extract,node_raw toneBlue
class node_loader,node_staging,node_analytics_tables toneAmber
class node_airflow,node_jolpica toneMint
class node_quality,node_standings toneRose
class node_operator toneIndigo
```

## Components

### Ingestion

- [Pipeline configuration](https://github.com/federicolaco/f1-data-pipeline/blob/main/src/config.py): `src/config.py`
- [Season extractor](https://github.com/federicolaco/f1-data-pipeline/blob/main/src/extract/jolpica_client.py): `src/extract/jolpica_client.py`
- [Raw JSON files](https://github.com/federicolaco/f1-data-pipeline/tree/main/data/raw): `data/raw`

### Warehouse

- [Postgres loader](https://github.com/federicolaco/f1-data-pipeline/blob/main/src/load/postgres_loader.py): `src/load/postgres_loader.py`
- [Staging tables](https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/01_staging.sql): `sql/01_staging.sql`
- [Analytics tables](https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/02_analytics.sql): `sql/02_analytics.sql`

### Orchestration

- [Airflow DAG](https://github.com/federicolaco/f1-data-pipeline/blob/main/dags/f1_pipeline.py): `dags/f1_pipeline.py`

### Quality and queries

- [Data quality checks](https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/03_checks.sql): `sql/03_checks.sql`
- [Driver standings query](https://github.com/federicolaco/f1-data-pipeline/blob/main/sql/04_driver_standings.sql): `sql/04_driver_standings.sql`

### Other components

- **Pipeline operator**
- **Jolpica API**

## Connections

- Pipeline operator → Season extractor: runs manually (evidence: `README.md`)
- Pipeline operator → Airflow DAG: triggers (evidence: `README.md`)
- Airflow DAG → Season extractor: calls extract (evidence: `dags/f1_pipeline.py`)
- Airflow DAG → Postgres loader: calls load tasks (evidence: `dags/f1_pipeline.py`)
- Season extractor → Pipeline configuration: reads settings (evidence: `src/extract/jolpica_client.py`)
- Postgres loader → Pipeline configuration: reads settings (evidence: `src/load/postgres_loader.py`)
- Season extractor → Jolpica API: fetches season data (evidence: `src/extract/jolpica_client.py`)
- Season extractor → Raw JSON files: writes JSON (evidence: `src/extract/jolpica_client.py`)
- Postgres loader → Raw JSON files: reads JSON (evidence: `src/load/postgres_loader.py`)
- Postgres loader → Staging tables: loads records (evidence: `src/load/postgres_loader.py`)
- Postgres loader → Analytics tables: promotes staging data (evidence: `src/load/postgres_loader.py`)
- Data quality checks → Staging tables: checks data
- Data quality checks → Analytics tables: checks data
- Driver standings query → Analytics tables: queries standings (evidence: `README.md`)
