How ssp-data/practical-data-engineering works
This Dagster project collects ImmoScout24 listings for configured searches, filters out unchanged properties, downloads details for new or changed listings, and merges them into a Delta table on S3-compatible storage. A Dagster notebook operation then explores the resulting data. The README also describes Druid ingestion and Superset visualization, but the shown job does not invoke Druid ingestion; those capabilities are represented as separate integrations. The graph follows the sampled pipeline and module references; it does not assume unsampled configuration or notebook behavior.
Main entry points: pipelines.py Dagster job · helper_functions.py Delta table · solids_scraping.py Detail fetch and cache · solids_spark_delta.py Delta upsert
Components 20
Dagster workflow
- Dagster job
src/pipelines/real-estate/realestate/pipelines.py - Search fan-out
src/pipelines/real-estate/realestate/pipelines.py - Search criteria types
src/pipelines/real-estate/realestate/common/types_realestate.py
Listing collection
- Listing discovery
src/pipelines/real-estate/realestate/common/solids_scraping.py - Change detection
src/pipelines/real-estate/realestate/common/solids_spark_delta.py - Detail fetch and cache
src/pipelines/real-estate/realestate/common/solids_scraping.py - JSON compression helpers
src/pipelines/real-estate/realestate/common/helper_functions.py
Delta lake
- Existing property table
src/pipelines/real-estate/realestate/pipelines.py - Delta table
src/pipelines/real-estate/realestate/common/helper_functions.py - JSON flattening
src/pipelines/real-estate/realestate/common/solids_spark_delta.py - Delta upsert
src/pipelines/real-estate/realestate/common/solids_spark_delta.py - Storage coordinates
src/pipelines/real-estate/realestate/common/types.py
Analytics and serving
- Notebook exploration
src/pipelines/real-estate/realestate/common/solids_jupyter.py - Exploration notebook
src/pipelines/real-estate/realestate/notebooks/comprehensive-real-estate-data-exploration.ipynb - Druid ingestion
src/pipelines/real-estate/realestate/common/solids_druid.py
Other components
- Pipeline operator
- ImmoScout24
- MinIO / S3
- Apache Druid
- Apache Superset
Connections 22
- Pipeline operator → Dagster job: starts job
README.md - Dagster job → Search fan-out: fans out
src/pipelines/real-estate/realestate/pipelines.py - Search fan-out → Search criteria types: uses type
src/pipelines/real-estate/realestate/pipelines.py - Search fan-out → Listing discovery: runs searches
src/pipelines/real-estate/realestate/pipelines.py - Listing discovery → ImmoScout24: requests listings
src/pipelines/real-estate/realestate/common/solids_scraping.py - Search fan-out → Existing property table: loads table
src/pipelines/real-estate/realestate/pipelines.py - Existing property table → Delta table: reads table
src/pipelines/real-estate/realestate/pipelines.py - Delta table → MinIO / S3: reads and writes
src/pipelines/real-estate/realestate/common/helper_functions.py - Search fan-out → Change detection: filters listings
src/pipelines/real-estate/realestate/pipelines.py - Change detection → Existing property table: compares records
src/pipelines/real-estate/realestate/common/solids_spark_delta.py - Dagster job → Detail fetch and cache: fetches changed details
src/pipelines/real-estate/realestate/pipelines.py - Detail fetch and cache → ImmoScout24: requests details
src/pipelines/real-estate/realestate/common/solids_scraping.py - Detail fetch and cache → JSON compression helpers: compresses payload
src/pipelines/real-estate/realestate/common/solids_scraping.py - Dagster job → JSON flattening: flattens cached data
src/pipelines/real-estate/realestate/pipelines.py - Dagster job → Delta upsert: merges records
src/pipelines/real-estate/realestate/pipelines.py - Delta upsert → Delta table: upserts records
src/pipelines/real-estate/realestate/common/solids_spark_delta.py - Delta upsert → Storage coordinates: returns location
src/pipelines/real-estate/realestate/common/solids_spark_delta.py - Dagster job → Notebook exploration: runs exploration
src/pipelines/real-estate/realestate/pipelines.py - Notebook exploration → Exploration notebook: executes notebook
src/pipelines/real-estate/realestate/common/solids_jupyter.py - Delta table → Druid ingestion: supplies table
src/pipelines/real-estate/realestate/common/solids_druid.py - Druid ingestion → Apache Druid: submits ingestion
src/pipelines/real-estate/realestate/common/solids_druid.py - Apache Druid → Apache Superset: visualizes data
README.md