# shiningflash/data-engineering-practice-problems architecture

> Architecture diagram of the GitHub repository shiningflash/data-engineering-practice-problems, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shiningflash/data-engineering-practice-problems
- Repository: https://github.com/shiningflash/data-engineering-practice-problems

## Overview

This repository is a catalog of practical data-engineering exercises, not a shared application or pipeline. An engineer uses the problem index or browses a problem folder, reads its question, then studies or runs its solution. The graph highlights the README’s core scenarios and broader problem areas represented in the tree. Sampled solutions show streaming analytics, data cleaning, partner-file normalization, and event-schema validation; unsampled problem implementations are not inferred.

Main entry points: [`PROBLEMS.md`](https://github.com/shiningflash/data-engineering-practice-problems/blob/main/PROBLEMS.md) (Problem index), [`problems/001-log-file-error-analysis`](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/001-log-file-error-analysis) (Log analysis), [`problems`](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems) (Problem scenario), [`problems/001-log-file-error-analysis/solution.py`](https://github.com/shiningflash/data-engineering-practice-problems/blob/main/problems/001-log-file-error-analysis/solution.py) (Problem solution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_practice["Practice problems"]
  node_index["Problem index<br/>[PROBLEMS.md]"]
  node_scenario["Problem scenario"]
  node_solution["Problem solution<br/>[solution.py]"]
end

subgraph group_data_quality["Data quality"]
  node_data_cleaning["Data cleaning"]
  node_schema_validation["Schema validation"]
  node_partner_files["Partner files"]
end

subgraph group_analytics["Analytics"]
  node_log_analysis["Log analysis"]
  node_rolling_average["Rolling average"]
  node_query_performance["Query performance"]
end

subgraph group_platform["Platform topics"]
  node_streaming["Streaming systems"]
  node_pipeline_ops["Pipeline operations"]
  node_data_modeling["Data modeling"]
  node_storage["Storage formats"]
  node_observability["Data observability"]
  node_system_design["System design"]
end

node_engineer(("Engineer"))

node_engineer -->|"browses"| node_index
node_engineer -->|"opens folder"| node_scenario
node_index -->|"links to"| node_log_analysis
node_index -->|"links to"| node_rolling_average
node_index -->|"links to"| node_data_cleaning
node_index -->|"links to"| node_schema_validation
node_index -->|"links to"| node_partner_files
node_index -->|"links to"| node_streaming
node_index -->|"links to"| node_pipeline_ops
node_index -->|"links to"| node_data_modeling
node_index -->|"links to"| node_query_performance
node_index -->|"links to"| node_storage
node_index -->|"links to"| node_observability
node_index -->|"links to"| node_system_design
node_log_analysis -->|"provides solution"| node_solution

click node_index "https://github.com/shiningflash/data-engineering-practice-problems/blob/main/PROBLEMS.md"
click node_scenario "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems"
click node_log_analysis "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/001-log-file-error-analysis"
click node_rolling_average "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/002-rolling-average-of-sensor-readings"
click node_data_cleaning "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/003-transform-and-clean-raw-data-for-analytics"
click node_schema_validation "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/004-schema-evolution-and-validation-for-streaming-events"
click node_partner_files "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/005-merging-messy-csvs-from-multiple-partners"
click node_streaming "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/057-kafka-ordering-guarantee"
click node_pipeline_ops "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/032-inheriting-a-pipeline-no-one-owns"
click node_data_modeling "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/042-tracking-subscription-plan-history"
click node_query_performance "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/017-reading-an-explain-plan"
click node_storage "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/080-iceberg-schema-evolution-and-time-travel"
click node_observability "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/084-data-observability-freshness-volume-drift"
click node_system_design "https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/021-data-platform-for-an-electricity-retailer"
click node_solution "https://github.com/shiningflash/data-engineering-practice-problems/blob/main/problems/001-log-file-error-analysis/solution.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_index,node_scenario,node_solution toneBlue
class node_data_cleaning,node_schema_validation,node_partner_files toneAmber
class node_log_analysis,node_rolling_average,node_query_performance toneMint
class node_streaming,node_pipeline_ops,node_data_modeling,node_storage,node_observability,node_system_design toneRose
class node_engineer toneIndigo
```

## Components

### Practice problems

- [Problem index](https://github.com/shiningflash/data-engineering-practice-problems/blob/main/PROBLEMS.md): `PROBLEMS.md`
- [Problem scenario](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems): `problems`
- [Problem solution](https://github.com/shiningflash/data-engineering-practice-problems/blob/main/problems/001-log-file-error-analysis/solution.py): `problems/001-log-file-error-analysis/solution.py`

### Data quality

- [Data cleaning](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/003-transform-and-clean-raw-data-for-analytics): `problems/003-transform-and-clean-raw-data-for-analytics`
- [Schema validation](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/004-schema-evolution-and-validation-for-streaming-events): `problems/004-schema-evolution-and-validation-for-streaming-events`
- [Partner files](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/005-merging-messy-csvs-from-multiple-partners): `problems/005-merging-messy-csvs-from-multiple-partners`

### Analytics

- [Log analysis](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/001-log-file-error-analysis): `problems/001-log-file-error-analysis`
- [Rolling average](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/002-rolling-average-of-sensor-readings): `problems/002-rolling-average-of-sensor-readings`
- [Query performance](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/017-reading-an-explain-plan): `problems/017-reading-an-explain-plan`

### Platform topics

- [Streaming systems](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/057-kafka-ordering-guarantee): `problems/057-kafka-ordering-guarantee`
- [Pipeline operations](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/032-inheriting-a-pipeline-no-one-owns): `problems/032-inheriting-a-pipeline-no-one-owns`
- [Data modeling](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/042-tracking-subscription-plan-history): `problems/042-tracking-subscription-plan-history`
- [Storage formats](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/080-iceberg-schema-evolution-and-time-travel): `problems/080-iceberg-schema-evolution-and-time-travel`
- [Data observability](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/084-data-observability-freshness-volume-drift): `problems/084-data-observability-freshness-volume-drift`
- [System design](https://github.com/shiningflash/data-engineering-practice-problems/tree/main/problems/021-data-platform-for-an-electricity-retailer): `problems/021-data-platform-for-an-electricity-retailer`

### Other components

- **Engineer**

## Connections

- Engineer → Problem index: browses
- Engineer → Problem scenario: opens folder
- Problem index → Log analysis: links to
- Problem index → Rolling average: links to
- Problem index → Data cleaning: links to
- Problem index → Schema validation: links to
- Problem index → Partner files: links to
- Problem index → Streaming systems: links to
- Problem index → Pipeline operations: links to
- Problem index → Data modeling: links to
- Problem index → Query performance: links to
- Problem index → Storage formats: links to
- Problem index → Data observability: links to
- Problem index → System design: links to
- Log analysis → Problem solution: provides solution