# aswina1636/codeforces-data-lakehouse architecture

> Architecture diagram of the GitHub repository aswina1636/codeforces-data-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aswina1636/codeforces-data-lakehouse
- Repository: https://github.com/aswina1636/codeforces-data-lakehouse

## Overview

Codeforces Data Lakehouse ingests the problemset API, preserves timestamped raw JSON, converts it to Parquet, loads PostgreSQL staging tables, builds dbt warehouse models and analytics marts, and validates the results. Airflow schedules and sequences these steps; Power BI provides the documented dashboard layer. The graph distinguishes implementation-supported script calls from README-documented dbt and visualization flows; unsampled dbt model details are represented at their existing directories.

Main entry points: [`airflow/dags/codeforces_pipeline.py`](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/airflow/dags/codeforces_pipeline.py) (Airflow DAG), [`src/codeforces_lakehouse/ingestion/postgres_loader.py`](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/postgres_loader.py) (PostgreSQL warehouse), [`scripts/validate_pipeline.py`](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/validate_pipeline.py) (Pipeline validation), [`data/processed`](https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/data/processed) (Processed Parquet)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_api_client["API client"]
  node_settings["Settings<br/>[settings.py]"]
  node_ingest_runner["Ingestion runner"]
  node_raw_writer["Raw JSON writer<br/>[raw_writer.py]"]
end

subgraph group_lake["Data lake"]
  node_raw_json["Raw JSON"]
  node_process_runner["Processing runner"]
  node_processor["Parquet processor"]
  node_parquet["Processed Parquet"]
  node_query_parquet["Parquet queries<br/>[query_parquet.py]"]
end

subgraph group_warehouse["Warehouse"]
  node_load_runner["Load runner<br/>[load_postgres.py]"]
  node_loader["Postgres loader<br/>[postgres_loader.py]"]
  node_postgres[("PostgreSQL warehouse<br/>[postgres_loader.py]")]
end

subgraph group_ops["Orchestration and quality"]
  node_airflow["Airflow DAG"]
  node_validator["Pipeline validation"]
end

subgraph group_analytics["Analytics"]
  node_dbt_staging["Staging models"]
  node_dbt_warehouse["Warehouse models"]
  node_dbt_marts["Analytics marts"]
  node_powerbi["Power BI dashboard<br/>[Dashboard.pbix]"]
end

node_codeforces(("Codeforces API"))

node_api_client -->|"reads settings"| node_settings
node_ingest_runner -->|"requests problemset"| node_api_client
node_ingest_runner -->|"writes response"| node_raw_writer
node_raw_writer -->|"writes JSON"| node_raw_json
node_process_runner -->|"reads snapshot"| node_raw_json
node_process_runner -->|"runs processing"| node_processor
node_processor -->|"writes Parquet"| node_parquet
node_query_parquet -->|"queries files"| node_parquet
node_load_runner -->|"runs loader"| node_loader
node_loader -->|"reads files"| node_parquet
node_loader -->|"reads settings"| node_settings
node_loader -->|"loads staging"| node_postgres
node_postgres -->|"provides sources"| node_dbt_staging
node_dbt_staging -.->|"transforms data"| node_dbt_warehouse
node_dbt_warehouse -.->|"builds marts"| node_dbt_marts
node_dbt_marts -->|"writes marts"| node_postgres
node_airflow -->|"runs task"| node_ingest_runner
node_airflow -->|"runs task"| node_process_runner
node_airflow -->|"runs task"| node_load_runner
node_airflow -->|"runs dbt build"| node_dbt_marts
node_airflow -->|"runs validation"| node_validator
node_validator -->|"checks counts"| node_raw_json
node_validator -->|"checks counts"| node_parquet
node_validator -->|"checks tables"| node_postgres
node_postgres -->|"feeds dashboards"| node_powerbi
node_codeforces -->|"serves API"| node_api_client

click node_api_client "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/api/codeforces_client.py"
click node_settings "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/config/settings.py"
click node_ingest_runner "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/ingest_problemset.py"
click node_raw_writer "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/raw_writer.py"
click node_raw_json "https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/data/raw"
click node_process_runner "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/process_problemset.py"
click node_processor "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/problemset_processor.py"
click node_parquet "https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/data/processed"
click node_query_parquet "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/query_parquet.py"
click node_load_runner "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/load_postgres.py"
click node_loader "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/postgres_loader.py"
click node_postgres "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/postgres_loader.py"
click node_dbt_staging "https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/dbt/codeforces_warehouse/models/staging"
click node_dbt_warehouse "https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/dbt/codeforces_warehouse/models/warehouse"
click node_dbt_marts "https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/dbt/codeforces_warehouse/models/marts"
click node_airflow "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/airflow/dags/codeforces_pipeline.py"
click node_validator "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/validate_pipeline.py"
click node_powerbi "https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/Dashboard.pbix"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_client,node_settings,node_ingest_runner,node_raw_writer toneBlue
class node_raw_json,node_process_runner,node_processor,node_parquet,node_query_parquet toneAmber
class node_load_runner,node_loader,node_postgres,node_codeforces toneMint
class node_airflow,node_validator toneRose
class node_dbt_staging,node_dbt_warehouse,node_dbt_marts,node_powerbi toneIndigo
```

## Components

### Ingestion

- [API client](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/api/codeforces_client.py): `src/codeforces_lakehouse/api/codeforces_client.py`
- [Settings](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/config/settings.py): `src/codeforces_lakehouse/config/settings.py`
- [Ingestion runner](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/ingest_problemset.py): `scripts/ingest_problemset.py`
- [Raw JSON writer](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/raw_writer.py): `src/codeforces_lakehouse/ingestion/raw_writer.py`

### Data lake

- [Raw JSON](https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/data/raw): `data/raw`
- [Processing runner](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/process_problemset.py): `scripts/process_problemset.py`
- [Parquet processor](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/problemset_processor.py): `src/codeforces_lakehouse/ingestion/problemset_processor.py`
- [Processed Parquet](https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/data/processed): `data/processed`
- [Parquet queries](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/query_parquet.py): `scripts/query_parquet.py`

### Warehouse

- [Load runner](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/load_postgres.py): `scripts/load_postgres.py`
- [Postgres loader](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/postgres_loader.py): `src/codeforces_lakehouse/ingestion/postgres_loader.py`
- [PostgreSQL warehouse](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/src/codeforces_lakehouse/ingestion/postgres_loader.py): `src/codeforces_lakehouse/ingestion/postgres_loader.py`

### Orchestration and quality

- [Airflow DAG](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/airflow/dags/codeforces_pipeline.py): `airflow/dags/codeforces_pipeline.py`
- [Pipeline validation](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/scripts/validate_pipeline.py): `scripts/validate_pipeline.py`

### Analytics

- [Staging models](https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/dbt/codeforces_warehouse/models/staging): `dbt/codeforces_warehouse/models/staging`
- [Warehouse models](https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/dbt/codeforces_warehouse/models/warehouse): `dbt/codeforces_warehouse/models/warehouse`
- [Analytics marts](https://github.com/aswina1636/codeforces-data-lakehouse/tree/main/dbt/codeforces_warehouse/models/marts): `dbt/codeforces_warehouse/models/marts`
- [Power BI dashboard](https://github.com/aswina1636/codeforces-data-lakehouse/blob/main/Dashboard.pbix): `Dashboard.pbix`

### Other components

- **Codeforces API**

## Connections

- API client → Settings: reads settings (evidence: `src/codeforces_lakehouse/api/codeforces_client.py`)
- Ingestion runner → API client: requests problemset (evidence: `scripts/ingest_problemset.py`)
- Ingestion runner → Raw JSON writer: writes response (evidence: `scripts/ingest_problemset.py`)
- Raw JSON writer → Raw JSON: writes JSON
- Processing runner → Raw JSON: reads snapshot (evidence: `scripts/process_problemset.py`)
- Processing runner → Parquet processor: runs processing (evidence: `scripts/process_problemset.py`)
- Parquet processor → Processed Parquet: writes Parquet (evidence: `src/codeforces_lakehouse/ingestion/problemset_processor.py`)
- Parquet queries → Processed Parquet: queries files (evidence: `scripts/query_parquet.py`)
- Load runner → Postgres loader: runs loader (evidence: `scripts/load_postgres.py`)
- Postgres loader → Processed Parquet: reads files (evidence: `src/codeforces_lakehouse/ingestion/postgres_loader.py`)
- Postgres loader → Settings: reads settings (evidence: `src/codeforces_lakehouse/ingestion/postgres_loader.py`)
- Postgres loader → PostgreSQL warehouse: loads staging (evidence: `src/codeforces_lakehouse/ingestion/postgres_loader.py`)
- PostgreSQL warehouse → Staging models: provides sources (evidence: `README.md`)
- Staging models → Warehouse models: transforms data
- Warehouse models → Analytics marts: builds marts
- Analytics marts → PostgreSQL warehouse: writes marts (evidence: `README.md`)
- Airflow DAG → Ingestion runner: runs task (evidence: `airflow/dags/codeforces_pipeline.py`)
- Airflow DAG → Processing runner: runs task (evidence: `airflow/dags/codeforces_pipeline.py`)
- Airflow DAG → Load runner: runs task (evidence: `airflow/dags/codeforces_pipeline.py`)
- Airflow DAG → Analytics marts: runs dbt build (evidence: `airflow/dags/codeforces_pipeline.py`)
- Airflow DAG → Pipeline validation: runs validation (evidence: `airflow/dags/codeforces_pipeline.py`)
- Pipeline validation → Raw JSON: checks counts (evidence: `scripts/validate_pipeline.py`)
- Pipeline validation → Processed Parquet: checks counts (evidence: `scripts/validate_pipeline.py`)
- Pipeline validation → PostgreSQL warehouse: checks tables (evidence: `scripts/validate_pipeline.py`)
- PostgreSQL warehouse → Power BI dashboard: feeds dashboards (evidence: `README.md`)
- Codeforces API → API client: serves API (evidence: `src/codeforces_lakehouse/api/codeforces_client.py`)
