# apress/practical-data-engineering-with-apache-projects architecture

> Architecture diagram of the GitHub repository apress/practical-data-engineering-with-apache-projects, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apress/practical-data-engineering-with-apache-projects
- Repository: https://github.com/apress/practical-data-engineering-with-apache-projects

## Overview

This repository is a book’s collection of hands-on OneShop data-engineering projects, not one deployed application. Its main user-to-result workflows are chapter-specific: batch ingestion and lakehouse analytics, streaming/CDC and anomaly detection, and ML-powered recommendations and review search. The graph preserves those distinct stages and their documented outputs. Sampled code supports the batch, segmentation, flash-sale dashboard, recommendation, and semantic-search links; other chapter capabilities are represented from the README, with uncertain implementation wiring omitted.

Main entry points: [`chapter-02`](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-02) (Iceberg foundation), [`chapter-05/airflow/dags/user_engagement_segments_dag.py`](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py) (Engagement segments), [`chapter-10/streamlit/embedding_generator.py`](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/streamlit/embedding_generator.py) (Embedding generation), [`chapter-10/streamlit/app.py`](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/streamlit/app.py) (Semantic review search)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_lakehouse["Lakehouse analytics"]
  node_iceberg_setup["Iceberg foundation"]
  node_batch_source[("Operational data")]
  node_batch_ingest["Bronze ingestion<br/>[postgres_loader.py]"]
  node_pageview_ingest["Pageview ingestion<br/>[minio_loader.py]"]
  node_object_store[("MinIO objects")]
  node_transform["Silver transforms"]
  node_gold_bi[("Gold BI tables")]
  node_dashboards["BI dashboards"]
  node_segments["Engagement segments"]
  node_segment_export["CSV export"]
  node_segment_storage[("Segment files")]
  node_segment_email["Email reminder"]
end

subgraph group_streaming["Streaming systems"]
  node_cdc["Inventory CDC"]
  node_search_index[("OpenSearch index")]
  node_flash_data[("Flash-sale purchases")]
  node_flash_stream["Kafka analytics<br/>[compose.yaml]"]
  node_clickhouse[("ClickHouse analytics")]
  node_flash_dashboard["Flash-sale dashboard<br/>[app.py]"]
  node_login_events["Login events<br/>[generate_load.py]"]
  node_anomaly["Login anomaly detection<br/>[insert-jobs.sql]"]
end

subgraph group_ml["AI and machine learning"]
  node_feature_pipeline["Feature engineering"]
  node_feature_store[("Gold features")]
  node_recommender["ALS training and serving"]
  node_recommendation_api["Recommendation API<br/>[app.py]"]
  node_reviews[("Customer reviews<br/>[reviews.sql]")]
  node_embedding_job["Embedding generation"]
  node_review_search["Semantic review search<br/>[app.py]"]
end

node_learner(("Engineer or analyst"))
node_email_service["Email service"]
node_embedding_model["Sentence-transformer model"]

node_learner -->|"sets up"| node_iceberg_setup
node_batch_source -->|"supplies rows"| node_batch_ingest
node_batch_ingest -->|"writes bronze"| node_iceberg_setup
node_object_store -->|"provides objects"| node_pageview_ingest
node_pageview_ingest -->|"writes bronze"| node_iceberg_setup
node_transform -->|"reads and writes"| node_iceberg_setup
node_iceberg_setup -->|"provides lakehouse tables"| node_gold_bi
node_gold_bi -->|"queries"| node_dashboards
node_learner -->|"views"| node_dashboards
node_segments -->|"queries and segments"| node_gold_bi
node_segments -->|"exports CSV"| node_segment_export
node_segment_export -->|"uploads"| node_segment_storage
node_segments -->|"sends reminder"| node_email_service
node_email_service -->|"delivers"| node_segment_email
node_batch_source -->|"emits changes"| node_cdc
node_cdc -->|"updates index"| node_search_index
node_flash_data -->|"publishes events"| node_flash_stream
node_flash_stream -->|"loads analytics"| node_clickhouse
node_flash_dashboard -->|"queries"| node_clickhouse
node_learner -->|"views"| node_flash_dashboard
node_login_events -->|"streams events"| node_anomaly
node_iceberg_setup -->|"supplies silver tables"| node_feature_pipeline
node_feature_pipeline -->|"writes features"| node_feature_store
node_feature_store -->|"provides features"| node_recommender
node_recommender -->|"stores recommendations"| node_recommendation_api
node_learner -->|"requests results"| node_recommendation_api
node_recommendation_api -->|"returns recommendations"| node_learner
node_reviews -->|"provides review text"| node_embedding_job
node_embedding_job -->|"encodes text"| node_embedding_model
node_embedding_job -->|"writes embeddings"| node_reviews
node_review_search -->|"encodes query"| node_embedding_model
node_review_search -->|"searches vectors"| node_reviews
node_learner -->|"searches reviews"| node_review_search

click node_iceberg_setup "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-02"
click node_batch_source "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-03/postgres"
click node_batch_ingest "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-03/spark/scripts/postgres_loader.py"
click node_pageview_ingest "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-03/spark/scripts/minio_loader.py"
click node_transform "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-03/spark/scripts/bronze_to_silver_transformer.py"
click node_gold_bi "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-04/trino"
click node_dashboards "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-04/superset"
click node_segments "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py"
click node_segment_export "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py"
click node_segment_email "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py"
click node_cdc "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-06/kafka-connect"
click node_flash_data "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-07/postgres"
click node_flash_stream "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-07/compose.yaml"
click node_clickhouse "https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-07/clickhouse"
click node_flash_dashboard "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-07/streamlit/app.py"
click node_login_events "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-08/simulator/generate_load.py"
click node_anomaly "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-08/flink-sql/insert-jobs.sql"
click node_feature_pipeline "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-09/spark/scripts/compute_features.py"
click node_recommender "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-09/spark/scripts/train_and_serve_als.py"
click node_recommendation_api "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-09/flask/app.py"
click node_reviews "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/reviews.sql"
click node_embedding_job "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/streamlit/embedding_generator.py"
click node_review_search "https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/streamlit/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_iceberg_setup,node_batch_source,node_batch_ingest,node_pageview_ingest,node_object_store,node_transform,node_gold_bi,node_dashboards,node_segments,node_segment_export,node_segment_storage,node_segment_email toneBlue
class node_cdc,node_search_index,node_flash_data,node_flash_stream,node_clickhouse,node_flash_dashboard,node_login_events,node_anomaly toneAmber
class node_feature_pipeline,node_feature_store,node_recommender,node_recommendation_api,node_reviews,node_embedding_job,node_review_search toneMint
class node_learner,node_email_service,node_embedding_model toneIndigo
```

## Components

### Lakehouse analytics

- [Iceberg foundation](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-02): `chapter-02`
- [Operational data](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-03/postgres): `chapter-03/postgres`
- [Bronze ingestion](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-03/spark/scripts/postgres_loader.py): `chapter-03/spark/scripts/postgres_loader.py`
- [Pageview ingestion](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-03/spark/scripts/minio_loader.py): `chapter-03/spark/scripts/minio_loader.py`
- **MinIO objects**
- [Silver transforms](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-03/spark/scripts/bronze_to_silver_transformer.py): `chapter-03/spark/scripts/bronze_to_silver_transformer.py`
- [Gold BI tables](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-04/trino): `chapter-04/trino`
- [BI dashboards](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-04/superset): `chapter-04/superset`
- [Engagement segments](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py): `chapter-05/airflow/dags/user_engagement_segments_dag.py`
- [CSV export](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py): `chapter-05/airflow/dags/user_engagement_segments_dag.py`
- **Segment files**
- [Email reminder](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-05/airflow/dags/user_engagement_segments_dag.py): `chapter-05/airflow/dags/user_engagement_segments_dag.py`

### Streaming systems

- [Inventory CDC](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-06/kafka-connect): `chapter-06/kafka-connect`
- **OpenSearch index**
- [Flash-sale purchases](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-07/postgres): `chapter-07/postgres`
- [Kafka analytics](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-07/compose.yaml): `chapter-07/compose.yaml`
- [ClickHouse analytics](https://github.com/apress/practical-data-engineering-with-apache-projects/tree/main/chapter-07/clickhouse): `chapter-07/clickhouse`
- [Flash-sale dashboard](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-07/streamlit/app.py): `chapter-07/streamlit/app.py`
- [Login events](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-08/simulator/generate_load.py): `chapter-08/simulator/generate_load.py`
- [Login anomaly detection](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-08/flink-sql/insert-jobs.sql): `chapter-08/flink-sql/insert-jobs.sql`

### AI and machine learning

- [Feature engineering](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-09/spark/scripts/compute_features.py): `chapter-09/spark/scripts/compute_features.py`
- **Gold features**
- [ALS training and serving](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-09/spark/scripts/train_and_serve_als.py): `chapter-09/spark/scripts/train_and_serve_als.py`
- [Recommendation API](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-09/flask/app.py): `chapter-09/flask/app.py`
- [Customer reviews](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/reviews.sql): `chapter-10/reviews.sql`
- [Embedding generation](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/streamlit/embedding_generator.py): `chapter-10/streamlit/embedding_generator.py`
- [Semantic review search](https://github.com/apress/practical-data-engineering-with-apache-projects/blob/main/chapter-10/streamlit/app.py): `chapter-10/streamlit/app.py`

### Other components

- **Engineer or analyst**
- **Email service**
- **Sentence-transformer model**

## Connections

- Engineer or analyst → Iceberg foundation: sets up
- Operational data → Bronze ingestion: supplies rows
- Bronze ingestion → Iceberg foundation: writes bronze
- MinIO objects → Pageview ingestion: provides objects
- Pageview ingestion → Iceberg foundation: writes bronze
- Silver transforms → Iceberg foundation: reads and writes
- Iceberg foundation → Gold BI tables: provides lakehouse tables
- Gold BI tables → BI dashboards: queries
- Engineer or analyst → BI dashboards: views
- Engagement segments → Gold BI tables: queries and segments
- Engagement segments → CSV export: exports CSV
- CSV export → Segment files: uploads
- Engagement segments → Email service: sends reminder
- Email service → Email reminder: delivers
- Operational data → Inventory CDC: emits changes
- Inventory CDC → OpenSearch index: updates index
- Flash-sale purchases → Kafka analytics: publishes events
- Kafka analytics → ClickHouse analytics: loads analytics
- Flash-sale dashboard → ClickHouse analytics: queries
- Engineer or analyst → Flash-sale dashboard: views
- Login events → Login anomaly detection: streams events
- Iceberg foundation → Feature engineering: supplies silver tables
- Feature engineering → Gold features: writes features
- Gold features → ALS training and serving: provides features
- ALS training and serving → Recommendation API: stores recommendations
- Engineer or analyst → Recommendation API: requests results
- Recommendation API → Engineer or analyst: returns recommendations
- Customer reviews → Embedding generation: provides review text
- Embedding generation → Sentence-transformer model: encodes text
- Embedding generation → Customer reviews: writes embeddings
- Semantic review search → Sentence-transformer model: encodes query
- Semantic review search → Customer reviews: searches vectors
- Engineer or analyst → Semantic review search: searches reviews