# ssp-data/practical-data-engineering architecture

> Architecture diagram of the GitHub repository ssp-data/practical-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ssp-data/practical-data-engineering
- Repository: https://github.com/ssp-data/practical-data-engineering

## Overview

This Dagster project collects ImmoScout24 listings for configured searches, filters out unchanged properties, downloads details for new or changed listings, and merges them into a Delta table on S3-compatible storage. A Dagster notebook operation then explores the resulting data. The README also describes Druid ingestion and Superset visualization, but the shown job does not invoke Druid ingestion; those capabilities are represented as separate integrations. The graph follows the sampled pipeline and module references; it does not assume unsampled configuration or notebook behavior.

Main entry points: [`src/pipelines/real-estate/realestate/pipelines.py`](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py) (Dagster job), [`src/pipelines/real-estate/realestate/common/helper_functions.py`](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/helper_functions.py) (Delta table), [`src/pipelines/real-estate/realestate/common/solids_scraping.py`](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_scraping.py) (Detail fetch and cache), [`src/pipelines/real-estate/realestate/common/solids_spark_delta.py`](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py) (Delta upsert)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Dagster workflow"]
  node_dagster["Dagster job<br/>[pipelines.py]"]
  node_searches["Search fan-out<br/>[pipelines.py]"]
  node_criteria["Search criteria types"]
end

subgraph group_collection["Listing collection"]
  node_discovery["Listing discovery<br/>[solids_scraping.py]"]
  node_cdc["Change detection"]
  node_cache["Detail fetch and cache<br/>[solids_scraping.py]"]
  node_jsonhelpers["JSON compression helpers"]
end

subgraph group_lake["Delta lake"]
  node_propertytable["Existing property table<br/>[pipelines.py]"]
  node_delta[("Delta table")]
  node_flatten["JSON flattening"]
  node_merge["Delta upsert"]
  node_coordinates["Storage coordinates<br/>[types.py]"]
end

subgraph group_analytics["Analytics and serving"]
  node_explore["Notebook exploration<br/>[solids_jupyter.py]"]
  node_notebook["Exploration notebook"]
  node_druid_ingest["Druid ingestion<br/>[solids_druid.py]"]
end

node_engineer(("Pipeline operator"))
node_immo(("ImmoScout24"))
node_minio(("MinIO / S3"))
node_druid(("Apache Druid"))
node_superset(("Apache Superset"))

node_engineer -->|"starts job"| node_dagster
node_dagster -->|"fans out"| node_searches
node_searches -->|"uses type"| node_criteria
node_searches -->|"runs searches"| node_discovery
node_discovery -->|"requests listings"| node_immo
node_searches -->|"loads table"| node_propertytable
node_propertytable -->|"reads table"| node_delta
node_delta -->|"reads and writes"| node_minio
node_searches -->|"filters listings"| node_cdc
node_cdc -->|"compares records"| node_propertytable
node_dagster -->|"fetches changed details"| node_cache
node_cache -->|"requests details"| node_immo
node_cache -->|"compresses payload"| node_jsonhelpers
node_dagster -->|"flattens cached data"| node_flatten
node_dagster -->|"merges records"| node_merge
node_merge -->|"upserts records"| node_delta
node_merge -->|"returns location"| node_coordinates
node_dagster -->|"runs exploration"| node_explore
node_explore -->|"executes notebook"| node_notebook
node_delta -.->|"supplies table"| node_druid_ingest
node_druid_ingest -->|"submits ingestion"| node_druid
node_druid -.->|"visualizes data"| node_superset

click node_dagster "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py"
click node_searches "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py"
click node_criteria "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/types_realestate.py"
click node_discovery "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_scraping.py"
click node_cdc "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py"
click node_propertytable "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py"
click node_delta "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/helper_functions.py"
click node_cache "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_scraping.py"
click node_jsonhelpers "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/helper_functions.py"
click node_flatten "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py"
click node_merge "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py"
click node_coordinates "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/types.py"
click node_explore "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_jupyter.py"
click node_notebook "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/notebooks/comprehensive-real-estate-data-exploration.ipynb"
click node_druid_ingest "https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_druid.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dagster,node_searches,node_criteria toneBlue
class node_discovery,node_cdc,node_cache,node_jsonhelpers toneAmber
class node_propertytable,node_delta,node_flatten,node_merge,node_coordinates toneMint
class node_explore,node_notebook,node_druid_ingest toneRose
class node_engineer,node_immo,node_minio,node_druid,node_superset toneIndigo
```

## Components

### Dagster workflow

- [Dagster job](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py): `src/pipelines/real-estate/realestate/pipelines.py`
- [Search fan-out](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py): `src/pipelines/real-estate/realestate/pipelines.py`
- [Search criteria types](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/types_realestate.py): `src/pipelines/real-estate/realestate/common/types_realestate.py`

### Listing collection

- [Listing discovery](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_scraping.py): `src/pipelines/real-estate/realestate/common/solids_scraping.py`
- [Change detection](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py): `src/pipelines/real-estate/realestate/common/solids_spark_delta.py`
- [Detail fetch and cache](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_scraping.py): `src/pipelines/real-estate/realestate/common/solids_scraping.py`
- [JSON compression helpers](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/helper_functions.py): `src/pipelines/real-estate/realestate/common/helper_functions.py`

### Delta lake

- [Existing property table](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/pipelines.py): `src/pipelines/real-estate/realestate/pipelines.py`
- [Delta table](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/helper_functions.py): `src/pipelines/real-estate/realestate/common/helper_functions.py`
- [JSON flattening](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py): `src/pipelines/real-estate/realestate/common/solids_spark_delta.py`
- [Delta upsert](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_spark_delta.py): `src/pipelines/real-estate/realestate/common/solids_spark_delta.py`
- [Storage coordinates](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/types.py): `src/pipelines/real-estate/realestate/common/types.py`

### Analytics and serving

- [Notebook exploration](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_jupyter.py): `src/pipelines/real-estate/realestate/common/solids_jupyter.py`
- [Exploration notebook](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/notebooks/comprehensive-real-estate-data-exploration.ipynb): `src/pipelines/real-estate/realestate/notebooks/comprehensive-real-estate-data-exploration.ipynb`
- [Druid ingestion](https://github.com/ssp-data/practical-data-engineering/blob/main/src/pipelines/real-estate/realestate/common/solids_druid.py): `src/pipelines/real-estate/realestate/common/solids_druid.py`

### Other components

- **Pipeline operator**
- **ImmoScout24**
- **MinIO / S3**
- **Apache Druid**
- **Apache Superset**

## Connections

- Pipeline operator → Dagster job: starts job (evidence: `README.md`)
- Dagster job → Search fan-out: fans out (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Search fan-out → Search criteria types: uses type (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Search fan-out → Listing discovery: runs searches (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Listing discovery → ImmoScout24: requests listings (evidence: `src/pipelines/real-estate/realestate/common/solids_scraping.py`)
- Search fan-out → Existing property table: loads table (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Existing property table → Delta table: reads table (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Delta table → MinIO / S3: reads and writes (evidence: `src/pipelines/real-estate/realestate/common/helper_functions.py`)
- Search fan-out → Change detection: filters listings (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Change detection → Existing property table: compares records (evidence: `src/pipelines/real-estate/realestate/common/solids_spark_delta.py`)
- Dagster job → Detail fetch and cache: fetches changed details (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Detail fetch and cache → ImmoScout24: requests details (evidence: `src/pipelines/real-estate/realestate/common/solids_scraping.py`)
- Detail fetch and cache → JSON compression helpers: compresses payload (evidence: `src/pipelines/real-estate/realestate/common/solids_scraping.py`)
- Dagster job → JSON flattening: flattens cached data (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Dagster job → Delta upsert: merges records (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Delta upsert → Delta table: upserts records (evidence: `src/pipelines/real-estate/realestate/common/solids_spark_delta.py`)
- Delta upsert → Storage coordinates: returns location (evidence: `src/pipelines/real-estate/realestate/common/solids_spark_delta.py`)
- Dagster job → Notebook exploration: runs exploration (evidence: `src/pipelines/real-estate/realestate/pipelines.py`)
- Notebook exploration → Exploration notebook: executes notebook (evidence: `src/pipelines/real-estate/realestate/common/solids_jupyter.py`)
- Delta table → Druid ingestion: supplies table (evidence: `src/pipelines/real-estate/realestate/common/solids_druid.py`)
- Druid ingestion → Apache Druid: submits ingestion (evidence: `src/pipelines/real-estate/realestate/common/solids_druid.py`)
- Apache Druid → Apache Superset: visualizes data (evidence: `README.md`)
