How darshilparmar/stock-market-kafka-data-engineering-project works
Project Type & Purpose
• This project is an end-to-end data engineering pipeline focused on handling real-time stock market data. It leverages Apache Kafka for streaming, AWS services (such as S3, Athena, Glue, EC2) for storage, processing, and cataloging, and Python for implementation.
• The README clearly outlines that the project ingests stock market data with Kafka and further processes it via AWS tools.
File Structure Analysis
• The file tree shows two Jupyter notebooks (KafkaConsumer.ipynb and KafkaProducer.ipynb) which indicate separate roles in the pipeline: one for consuming Kafka streams and one for producing messages.
• presence of command_kafka.txt suggests a script or command line instructions for Kafka operations.
• The indexProcessed.csv file is likely the dataset used within the project.
Identifying Main Components
a. Data Producers:
• Kafka Producer (as implemented in KafkaProducer.ipynb)
b. Message Broker:
• Apache Kafka as the central component that handles real-time data streaming.
c. Data Consumers:
• Kafka Consumer (as implemented in KafkaConsumer.ipynb)
d. AWS Ecosystem Components:
• S3: For data storage.
• Athena: For query execution on the stored data.
• Glue Crawler & Glue Catalog: For data discovery and metadata management.
• EC2: Likely for executing parts of the processing or running additional services.
e. Data Source:
• The stock market dataset (indexProcessed.csv) that is either ingested or processed to simulate streaming data.
f. Auxiliary/Operational Elements:
• Command scripts (command_kafka.txt) that may manage or automate Kafka configurations.
Relationships & Interactions
• Data flows from the Data Source through the Kafka Producer into the Kafka topic.
• The Kafka Consumer then processes this streamed data.
• Processed data is then sent to AWS S3 for storage.
• AWS Glue (Crawler and Catalog) picks up the new data for cataloging.
• Athena queries operate on the cataloged data.
• EC2 instances might be hosting some of these processes or managing AWS service interactions.
Architectural Patterns & Technologies
• Core components like Apache Kafka and AWS components along with Python-based processing.
• Mention the use of Jupyter notebooks which serve as the development/testing interface for the Kafka components.