How katyayani09/azure-data-engineering-projects works
This repository is an Olympic-sports data engineering and analytics project. Its visible runtime pieces cover Kafka publishing, Azure Event Hubs consumption with checkpointing, and Databricks preprocessing/sample notebooks; the supplied CSVs provide Olympic athlete, coach, team, and medal data. The snippets do not establish that the CSVs feed the Kafka producer or that Kafka and Event Hubs are connected, so those links are omitted. The visualization directory is present but contains no listed implementation.
Main entry points: kafka_producer.py Kafka Producer · event_hub_consumer.py Event Hub Consumer
Components 8
Olympic Data
- Olympic CSVs
Olympic-Games-Trends-and-Global-Sports-Analytics/sample_data/raw_data
Event Ingestion
- Kafka Producer
Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/kafka_producer.py - Event Hub Consumer
Olympic-Games-Trends-and-Global-Sports-Analytics/src/ingestion/event_hub_consumer.py
Databricks Analytics
- Databricks Preprocessing
Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_preprocessing.dbc.ipynb - Databricks Sample
Olympic-Games-Trends-and-Global-Sports-Analytics/src/transformation/databricks_notebook_sample.dbc.ipynb - Visualization Area
Olympic-Games-Trends-and-Global-Sports-Analytics/src/visualization
Other components
- Kafka Broker
- Azure Event Hub
Connections 2
- Kafka Producer → Kafka Broker: publishes to
- Azure Event Hub → Event Hub Consumer: delivers events