# lifewitdata/programmatic-advertising-ctr-prediction-pipeline architecture

> Architecture diagram of the GitHub repository lifewitdata/programmatic-advertising-ctr-prediction-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline
- Repository: https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline

## Overview

This repository is a CTR-prediction project: it turns impression and click records into chronologically split, labeled datasets, then documents feature engineering and model selection. The principal implemented runtime path is CSV/PostgreSQL input → ETL → Parquet datasets; notebooks carry the downstream EDA, feature, and modeling work, with CatBoost selected by validation ROC-AUC. PostgreSQL loading and SQL analysis are separate database capabilities. The serving endpoint and real-time feature store described as future work are not implemented. Downstream notebook wiring is shown at the documented stage level; their source was not sampled.

Main entry points: [`src/etl/run_etl.py`](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/run_etl.py) (ETL runner), [`src/database/load_to_postgres.py`](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/database/load_to_postgres.py) (Postgres loader), [`src/etl/extract.py`](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/extract.py) (Data extraction), [`notebooks/feature_engineering.ipynb`](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/feature_engineering.ipynb) (Feature engineering)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_storage["Data storage"]
  node_loader["Postgres loader"]
  node_schema["Postgres schema<br/>[schema.sql]"]
  node_postgres[("PostgreSQL")]
  node_sql_queries["CTR queries"]
end

subgraph group_etl["ETL pipeline"]
  node_extract["Data extraction<br/>[extract.py]"]
  node_validate["Data validation<br/>[validate.py]"]
  node_clean["Data cleaning<br/>[clean.py]"]
  node_transform["Label and split<br/>[transform.py]"]
  node_orchestrator["ETL runner<br/>[run_etl.py]"]
  node_parquet["Processed datasets"]
  node_validation_report["Validation report"]
end

subgraph group_analysis["Data analysis"]
  node_eda["Exploratory analysis<br/>[eda.ipynb]"]
end

subgraph group_modeling["Feature and model"]
  node_features["Feature engineering"]
  node_modeling["Model comparison<br/>[modeling.ipynb]"]
  node_selected_model["Selected CatBoost<br/>[modeling.ipynb]"]
end

node_synthetic_csv["Synthetic CSVs"]

node_synthetic_csv -->|"bulk loads"| node_loader
node_loader -->|"applies schema"| node_schema
node_loader -->|"writes tables"| node_postgres
node_postgres -->|"queried by"| node_sql_queries
node_synthetic_csv -->|"reads CSVs"| node_extract
node_postgres -->|"reads tables"| node_extract
node_orchestrator -->|"calls"| node_extract
node_orchestrator -->|"calls"| node_validate
node_orchestrator -->|"calls"| node_clean
node_orchestrator -->|"calls"| node_transform
node_orchestrator -->|"writes datasets"| node_parquet
node_orchestrator -->|"writes report"| node_validation_report
node_parquet -->|"analyzed in"| node_eda
node_parquet -->|"engineered from"| node_features
node_features -->|"features used by"| node_modeling
node_modeling -->|"selects"| node_selected_model

click node_loader "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/database/load_to_postgres.py"
click node_schema "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/sql/schema.sql"
click node_sql_queries "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/sql/ctr_analysis_queries.sql"
click node_extract "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/extract.py"
click node_validate "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/validate.py"
click node_clean "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/clean.py"
click node_transform "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/transform.py"
click node_orchestrator "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/run_etl.py"
click node_eda "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/eda.ipynb"
click node_features "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/feature_engineering.ipynb"
click node_modeling "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/modeling.ipynb"
click node_selected_model "https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/modeling.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_loader,node_schema,node_postgres,node_sql_queries toneBlue
class node_extract,node_validate,node_clean,node_transform,node_orchestrator,node_parquet,node_validation_report toneAmber
class node_eda toneMint
class node_features,node_modeling,node_selected_model toneRose
class node_synthetic_csv toneIndigo
```

## Components

### Data storage

- [Postgres loader](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/database/load_to_postgres.py): `src/database/load_to_postgres.py`
- [Postgres schema](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/sql/schema.sql): `sql/schema.sql`
- **PostgreSQL**
- [CTR queries](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/sql/ctr_analysis_queries.sql): `sql/ctr_analysis_queries.sql`

### ETL pipeline

- [Data extraction](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/extract.py): `src/etl/extract.py`
- [Data validation](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/validate.py): `src/etl/validate.py`
- [Data cleaning](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/clean.py): `src/etl/clean.py`
- [Label and split](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/transform.py): `src/etl/transform.py`
- [ETL runner](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/src/etl/run_etl.py): `src/etl/run_etl.py`
- **Processed datasets**
- **Validation report**

### Data analysis

- [Exploratory analysis](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/eda.ipynb): `notebooks/eda.ipynb`

### Feature and model

- [Feature engineering](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/feature_engineering.ipynb): `notebooks/feature_engineering.ipynb`
- [Model comparison](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/modeling.ipynb): `notebooks/modeling.ipynb`
- [Selected CatBoost](https://github.com/lifewitdata/programmatic-advertising-ctr-prediction-pipeline/blob/main/notebooks/modeling.ipynb): `notebooks/modeling.ipynb`

### Other components

- **Synthetic CSVs**

## Connections

- Synthetic CSVs → Postgres loader: bulk loads
- Postgres loader → Postgres schema: applies schema
- Postgres loader → PostgreSQL: writes tables
- PostgreSQL → CTR queries: queried by
- Synthetic CSVs → Data extraction: reads CSVs
- PostgreSQL → Data extraction: reads tables
- ETL runner → Data extraction: calls
- ETL runner → Data validation: calls
- ETL runner → Data cleaning: calls
- ETL runner → Label and split: calls
- ETL runner → Processed datasets: writes datasets
- ETL runner → Validation report: writes report
- Processed datasets → Exploratory analysis: analyzed in
- Processed datasets → Feature engineering: engineered from
- Feature engineering → Model comparison: features used by
- Model comparison → Selected CatBoost: selects