How lifewitdata/programmatic-advertising-ctr-prediction-pipeline works
This repository is a CTR-prediction project: it turns impression and click records into chronologically split, labeled datasets, then documents feature engineering and model selection. The principal implemented runtime path is CSV/PostgreSQL input → ETL → Parquet datasets; notebooks carry the downstream EDA, feature, and modeling work, with CatBoost selected by validation ROC-AUC. PostgreSQL loading and SQL analysis are separate database capabilities. The serving endpoint and real-time feature store described as future work are not implemented. Downstream notebook wiring is shown at the documented stage level; their source was not sampled.
Main entry points: run_etl.py ETL runner · load_to_postgres.py Postgres loader · extract.py Data extraction · feature_engineering.ipynb Feature engineering
Components 16
Data storage
- Postgres loader
src/database/load_to_postgres.py - Postgres schema
sql/schema.sql - PostgreSQL
- CTR queries
sql/ctr_analysis_queries.sql
ETL pipeline
- Data extraction
src/etl/extract.py - Data validation
src/etl/validate.py - Data cleaning
src/etl/clean.py - Label and split
src/etl/transform.py - ETL runner
src/etl/run_etl.py - Processed datasets
- Validation report
Data analysis
- Exploratory analysis
notebooks/eda.ipynb
Feature and model
- Feature engineering
notebooks/feature_engineering.ipynb - Model comparison
notebooks/modeling.ipynb - Selected CatBoost
notebooks/modeling.ipynb
Other components
- Synthetic CSVs
Connections 16
- Synthetic CSVs → Postgres loader: bulk loads
- Postgres loader → Postgres schema: applies schema
- Postgres loader → PostgreSQL: writes tables
- PostgreSQL → CTR queries: queried by
- Synthetic CSVs → Data extraction: reads CSVs
- PostgreSQL → Data extraction: reads tables
- ETL runner → Data extraction: calls
- ETL runner → Data validation: calls
- ETL runner → Data cleaning: calls
- ETL runner → Label and split: calls
- ETL runner → Processed datasets: writes datasets
- ETL runner → Validation report: writes report
- Processed datasets → Exploratory analysis: analyzed in
- Processed datasets → Feature engineering: engineered from
- Feature engineering → Model comparison: features used by
- Model comparison → Selected CatBoost: selects