How shiningflash/data-engineering-practice-problems works
This repository is a catalog of practical data-engineering exercises, not a shared application or pipeline. An engineer uses the problem index or browses a problem folder, reads its question, then studies or runs its solution. The graph highlights the README’s core scenarios and broader problem areas represented in the tree. Sampled solutions show streaming analytics, data cleaning, partner-file normalization, and event-schema validation; unsampled problem implementations are not inferred.
Main entry points: PROBLEMS.md Problem index · 001-log-file-error-analysis Log analysis · problems Problem scenario · solution.py Problem solution
Components 16
Practice problems
- Problem index
PROBLEMS.md - Problem scenario
problems - Problem solution
problems/001-log-file-error-analysis/solution.py
Data quality
- Data cleaning
problems/003-transform-and-clean-raw-data-for-analytics - Schema validation
problems/004-schema-evolution-and-validation-for-streaming-events - Partner files
problems/005-merging-messy-csvs-from-multiple-partners
Analytics
- Log analysis
problems/001-log-file-error-analysis - Rolling average
problems/002-rolling-average-of-sensor-readings - Query performance
problems/017-reading-an-explain-plan
Platform topics
- Streaming systems
problems/057-kafka-ordering-guarantee - Pipeline operations
problems/032-inheriting-a-pipeline-no-one-owns - Data modeling
problems/042-tracking-subscription-plan-history - Storage formats
problems/080-iceberg-schema-evolution-and-time-travel - Data observability
problems/084-data-observability-freshness-volume-drift - System design
problems/021-data-platform-for-an-electricity-retailer
Other components
- Engineer
Connections 15
- Engineer → Problem index: browses
- Engineer → Problem scenario: opens folder
- Problem index → Log analysis: links to
- Problem index → Rolling average: links to
- Problem index → Data cleaning: links to
- Problem index → Schema validation: links to
- Problem index → Partner files: links to
- Problem index → Streaming systems: links to
- Problem index → Pipeline operations: links to
- Problem index → Data modeling: links to
- Problem index → Query performance: links to
- Problem index → Storage formats: links to
- Problem index → Data observability: links to
- Problem index → System design: links to
- Log analysis → Problem solution: provides solution