How aswina1636/codeforces-data-lakehouse works
Codeforces Data Lakehouse ingests the problemset API, preserves timestamped raw JSON, converts it to Parquet, loads PostgreSQL staging tables, builds dbt warehouse models and analytics marts, and validates the results. Airflow schedules and sequences these steps; Power BI provides the documented dashboard layer. The graph distinguishes implementation-supported script calls from README-documented dbt and visualization flows; unsampled dbt model details are represented at their existing directories.
Main entry points: codeforces_pipeline.py Airflow DAG · postgres_loader.py PostgreSQL warehouse · validate_pipeline.py Pipeline validation · processed Processed Parquet
Components 19
Ingestion
- API client
src/codeforces_lakehouse/api/codeforces_client.py - Settings
src/codeforces_lakehouse/config/settings.py - Ingestion runner
scripts/ingest_problemset.py - Raw JSON writer
src/codeforces_lakehouse/ingestion/raw_writer.py
Data lake
- Raw JSON
data/raw - Processing runner
scripts/process_problemset.py - Parquet processor
src/codeforces_lakehouse/ingestion/problemset_processor.py - Processed Parquet
data/processed - Parquet queries
scripts/query_parquet.py
Warehouse
- Load runner
scripts/load_postgres.py - Postgres loader
src/codeforces_lakehouse/ingestion/postgres_loader.py - PostgreSQL warehouse
src/codeforces_lakehouse/ingestion/postgres_loader.py
Orchestration and quality
- Airflow DAG
airflow/dags/codeforces_pipeline.py - Pipeline validation
scripts/validate_pipeline.py
Analytics
- Staging models
dbt/codeforces_warehouse/models/staging - Warehouse models
dbt/codeforces_warehouse/models/warehouse - Analytics marts
dbt/codeforces_warehouse/models/marts - Power BI dashboard
Dashboard.pbix
Other components
- Codeforces API
Connections 26
- API client → Settings: reads settings
src/codeforces_lakehouse/api/codeforces_client.py - Ingestion runner → API client: requests problemset
scripts/ingest_problemset.py - Ingestion runner → Raw JSON writer: writes response
scripts/ingest_problemset.py - Raw JSON writer → Raw JSON: writes JSON
- Processing runner → Raw JSON: reads snapshot
scripts/process_problemset.py - Processing runner → Parquet processor: runs processing
scripts/process_problemset.py - Parquet processor → Processed Parquet: writes Parquet
src/codeforces_lakehouse/ingestion/problemset_processor.py - Parquet queries → Processed Parquet: queries files
scripts/query_parquet.py - Load runner → Postgres loader: runs loader
scripts/load_postgres.py - Postgres loader → Processed Parquet: reads files
src/codeforces_lakehouse/ingestion/postgres_loader.py - Postgres loader → Settings: reads settings
src/codeforces_lakehouse/ingestion/postgres_loader.py - Postgres loader → PostgreSQL warehouse: loads staging
src/codeforces_lakehouse/ingestion/postgres_loader.py - PostgreSQL warehouse → Staging models: provides sources
README.md - Staging models → Warehouse models: transforms data
- Warehouse models → Analytics marts: builds marts
- Analytics marts → PostgreSQL warehouse: writes marts
README.md - Airflow DAG → Ingestion runner: runs task
airflow/dags/codeforces_pipeline.py - Airflow DAG → Processing runner: runs task
airflow/dags/codeforces_pipeline.py - Airflow DAG → Load runner: runs task
airflow/dags/codeforces_pipeline.py - Airflow DAG → Analytics marts: runs dbt build
airflow/dags/codeforces_pipeline.py - Airflow DAG → Pipeline validation: runs validation
airflow/dags/codeforces_pipeline.py - Pipeline validation → Raw JSON: checks counts
scripts/validate_pipeline.py - Pipeline validation → Processed Parquet: checks counts
scripts/validate_pipeline.py - Pipeline validation → PostgreSQL warehouse: checks tables
scripts/validate_pipeline.py - PostgreSQL warehouse → Power BI dashboard: feeds dashboards
README.md - Codeforces API → API client: serves API
src/codeforces_lakehouse/api/codeforces_client.py