# katyayani09/azure-data-engineering-projects architecture

> Architecture diagram of the GitHub repository katyayani09/azure-data-engineering-projects, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/katyayani09/azure-data-engineering-projects
- Repository: https://github.com/katyayani09/azure-data-engineering-projects

## Overview

This repository is an Olympic-sports data engineering and analytics project. Its visible runtime pieces cover Kafka publishing, Azure Event Hubs consumption with checkpointing, and Databricks preprocessing/sample notebooks; the supplied CSVs provide Olympic athlete, coach, team, and medal data. The snippets do not establish that the CSVs feed the Kafka producer or that Kafka and Event Hubs are connected, so those links are omitted. The visualization directory is present but contains no listed implementation.

Main entry points: [`Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/kafka_producer.py`](https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/kafka_producer.py) (Kafka Producer), [`Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/event_hub_consumer.py`](https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/event_hub_consumer.py) (Event Hub Consumer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Olympic Data"]
  node_csv["Olympic CSVs"]
end

subgraph group_ingestion["Event Ingestion"]
  node_producer["Kafka Producer<br/>[kafka_producer.py]"]
  node_consumer["Event Hub Consumer"]
end

subgraph group_analytics["Databricks Analytics"]
  node_preprocess["Databricks Preprocessing"]
  node_sample["Databricks Sample"]
  node_visualization["Visualization Area"]
end

node_kafka["Kafka Broker"]
node_eventhub["Azure Event Hub"]

node_producer -->|"publishes to"| node_kafka
node_eventhub -->|"delivers events"| node_consumer

click node_csv "https://github.com/katyayani09/azure-data-engineering-projects/tree/main/Olympic-Games-Trends-and-Global-Sports-Analytics/sample_data/raw_data"
click node_producer "https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/kafka_producer.py"
click node_consumer "https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/event_hub_consumer.py"
click node_preprocess "https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_preprocessing.dbc.ipynb"
click node_sample "https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_notebook_sample.dbc.ipynb"
click node_visualization "https://github.com/katyayani09/azure-data-engineering-projects/tree/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/visualization"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_csv toneBlue
class node_producer,node_consumer toneAmber
class node_preprocess,node_sample,node_visualization toneMint
class node_kafka,node_eventhub toneIndigo
```

## Components

### Olympic Data

- [Olympic CSVs](https://github.com/katyayani09/azure-data-engineering-projects/tree/main/Olympic-Games-Trends-and-Global-Sports-Analytics/sample_data/raw_data): `Olympic-Games-Trends-and-Global-Sports-Analytics/sample_data/raw_data`

### Event Ingestion

- [Kafka Producer](https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/kafka_producer.py): `Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/kafka_producer.py`
- [Event Hub Consumer](https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/event_hub_consumer.py): `Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/event_hub_consumer.py`

### Databricks Analytics

- [Databricks Preprocessing](https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_preprocessing.dbc.ipynb): `Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_preprocessing.dbc.ipynb`
- [Databricks Sample](https://github.com/katyayani09/azure-data-engineering-projects/blob/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_notebook_sample.dbc.ipynb): `Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_notebook_sample.dbc.ipynb`
- [Visualization Area](https://github.com/katyayani09/azure-data-engineering-projects/tree/main/Olympic-Games-Trends-and-Global-Sports-Analytics/src/visualization): `Olympic-Games-Trends-and-Global-Sports-Analytics/src/visualization`

### Other components

- **Kafka Broker**
- **Azure Event Hub**

## Connections

- Kafka Producer → Kafka Broker: publishes to
- Azure Event Hub → Event Hub Consumer: delivers events