How apress/practical-data-engineering-with-apache-projects works
This repository is a book’s collection of hands-on OneShop data-engineering projects, not one deployed application. Its main user-to-result workflows are chapter-specific: batch ingestion and lakehouse analytics, streaming/CDC and anomaly detection, and ML-powered recommendations and review search. The graph preserves those distinct stages and their documented outputs. Sampled code supports the batch, segmentation, flash-sale dashboard, recommendation, and semantic-search links; other chapter capabilities are represented from the README, with uncertain implementation wiring omitted.
Main entry points: chapter-02 Iceberg foundation · user_engagement_segments_dag.py Engagement segments · embedding_generator.py Embedding generation · app.py Semantic review search
Components 30
Lakehouse analytics
- Iceberg foundation
chapter-02 - Operational data
chapter-03/postgres - Bronze ingestion
chapter-03/spark/scripts/postgres_loader.py - Pageview ingestion
chapter-03/spark/scripts/minio_loader.py - MinIO objects
- Silver transforms
chapter-03/spark/scripts/bronze_to_silver_transformer.py - Gold BI tables
chapter-04/trino - BI dashboards
chapter-04/superset - Engagement segments
chapter-05/airflow/dags/user_engagement_segments_dag.py - CSV export
chapter-05/airflow/dags/user_engagement_segments_dag.py - Segment files
- Email reminder
chapter-05/airflow/dags/user_engagement_segments_dag.py
Streaming systems
- Inventory CDC
chapter-06/kafka-connect - OpenSearch index
- Flash-sale purchases
chapter-07/postgres - Kafka analytics
chapter-07/compose.yaml - ClickHouse analytics
chapter-07/clickhouse - Flash-sale dashboard
chapter-07/streamlit/app.py - Login events
chapter-08/simulator/generate_load.py - Login anomaly detection
chapter-08/flink-sql/insert-jobs.sql
AI and machine learning
- Feature engineering
chapter-09/spark/scripts/compute_features.py - Gold features
- ALS training and serving
chapter-09/spark/scripts/train_and_serve_als.py - Recommendation API
chapter-09/flask/app.py - Customer reviews
chapter-10/reviews.sql - Embedding generation
chapter-10/streamlit/embedding_generator.py - Semantic review search
chapter-10/streamlit/app.py
Other components
- Engineer or analyst
- Email service
- Sentence-transformer model
Connections 33
- Engineer or analyst → Iceberg foundation: sets up
- Operational data → Bronze ingestion: supplies rows
- Bronze ingestion → Iceberg foundation: writes bronze
- MinIO objects → Pageview ingestion: provides objects
- Pageview ingestion → Iceberg foundation: writes bronze
- Silver transforms → Iceberg foundation: reads and writes
- Iceberg foundation → Gold BI tables: provides lakehouse tables
- Gold BI tables → BI dashboards: queries
- Engineer or analyst → BI dashboards: views
- Engagement segments → Gold BI tables: queries and segments
- Engagement segments → CSV export: exports CSV
- CSV export → Segment files: uploads
- Engagement segments → Email service: sends reminder
- Email service → Email reminder: delivers
- Operational data → Inventory CDC: emits changes
- Inventory CDC → OpenSearch index: updates index
- Flash-sale purchases → Kafka analytics: publishes events
- Kafka analytics → ClickHouse analytics: loads analytics
- Flash-sale dashboard → ClickHouse analytics: queries
- Engineer or analyst → Flash-sale dashboard: views
- Login events → Login anomaly detection: streams events
- Iceberg foundation → Feature engineering: supplies silver tables
- Feature engineering → Gold features: writes features
- Gold features → ALS training and serving: provides features
- ALS training and serving → Recommendation API: stores recommendations
- Engineer or analyst → Recommendation API: requests results
- Recommendation API → Engineer or analyst: returns recommendations
- Customer reviews → Embedding generation: provides review text
- Embedding generation → Sentence-transformer model: encodes text
- Embedding generation → Customer reviews: writes embeddings
- Semantic review search → Sentence-transformer model: encodes query
- Semantic review search → Customer reviews: searches vectors
- Engineer or analyst → Semantic review search: searches reviews