# pashk3r/steam-games-etl-pipeline architecture

> Architecture diagram of the GitHub repository pashk3r/steam-games-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pashk3r/steam-games-etl-pipeline
- Repository: https://github.com/pashk3r/steam-games-etl-pipeline

## Overview

This repository is a Steam-games ETL application. The principal workflow starts in `main.py`, which creates a Spark session and invokes CSV extraction, PySpark transformation, then JDBC loading to PostgreSQL. Transformation covers the documented cleaning, date/owner splitting, and string-to-array conversions; loading serializes arrays for storage. Configuration supplies runtime settings and logging. Relationships are based on the provided source excerpts and README; no unsampled implementation is assumed.

Main entry points: [`main.py`](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/main.py) (Pipeline runner), [`src/extract.py`](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/extract.py) (CSV extraction), [`src/load.py`](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/load.py) (JDBC loading), [`src/config.py`](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/config.py) (Configuration and logging)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["ETL pipeline"]
  node_main["Pipeline runner<br/>[main.py]"]
  node_extract["CSV extraction<br/>[extract.py]"]
  node_transform["Data transformation<br/>[transform.py]"]
  node_load["JDBC loading<br/>[load.py]"]
end

subgraph group_support["Runtime support"]
  node_config["Configuration and logging<br/>[config.py]"]
  node_exceptions["ETL exceptions<br/>[exceptions.py]"]
end

node_user(("Operator"))
node_csv["Games CSV"]
node_postgres[("PostgreSQL")]

node_user -->|"runs"| node_main
node_main -->|"sets up"| node_config
node_main -->|"invokes"| node_extract
node_csv -->|"read and fix"| node_extract
node_extract -->|"returns DataFrame"| node_main
node_main -->|"invokes"| node_transform
node_transform -->|"returns DataFrame"| node_main
node_main -->|"invokes"| node_load
node_load -->|"writes via JDBC"| node_postgres
node_extract -->|"reads CSV path"| node_config
node_load -->|"reads DB settings"| node_config
node_main -->|"catches ETL errors"| node_exceptions

click node_main "https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/main.py"
click node_extract "https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/extract.py"
click node_transform "https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/transform.py"
click node_load "https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/load.py"
click node_config "https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/config.py"
click node_exceptions "https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/exceptions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_extract,node_transform,node_load toneBlue
class node_config,node_exceptions,node_postgres toneAmber
class node_user,node_csv toneIndigo
```

## Components

### ETL pipeline

- [Pipeline runner](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/main.py): `main.py`
- [CSV extraction](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/extract.py): `src/extract.py`
- [Data transformation](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/transform.py): `src/transform.py`
- [JDBC loading](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/load.py): `src/load.py`

### Runtime support

- [Configuration and logging](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/config.py): `src/config.py`
- [ETL exceptions](https://github.com/pashk3r/steam-games-etl-pipeline/blob/main/src/exceptions.py): `src/exceptions.py`

### Other components

- **Operator**
- **Games CSV**
- **PostgreSQL**

## Connections

- Operator → Pipeline runner: runs (evidence: `README.md`)
- Pipeline runner → Configuration and logging: sets up (evidence: `main.py`)
- Pipeline runner → CSV extraction: invokes (evidence: `main.py`)
- Games CSV → CSV extraction: read and fix (evidence: `src/extract.py`)
- CSV extraction → Pipeline runner: returns DataFrame (evidence: `main.py`)
- Pipeline runner → Data transformation: invokes (evidence: `main.py`)
- Data transformation → Pipeline runner: returns DataFrame (evidence: `main.py`)
- Pipeline runner → JDBC loading: invokes (evidence: `main.py`)
- JDBC loading → PostgreSQL: writes via JDBC (evidence: `src/load.py`)
- CSV extraction → Configuration and logging: reads CSV path (evidence: `src/extract.py`)
- JDBC loading → Configuration and logging: reads DB settings (evidence: `src/load.py`)
- Pipeline runner → ETL exceptions: catches ETL errors (evidence: `main.py`)
