How dhruveww/stock-market-real-time-analysis-with-kafka-and-aws works
This repository documents a real-time stock-data pipeline: a Python producer publishes simulated data to Kafka, a consumer writes it to S3, and Glue cataloging enables Athena queries and optional visualization. The notebooks and query file are present, but their contents were not provided, so component behavior and links below rely on README statements rather than inspected code; the CSV is documented as a simulated dataset, but its runtime use is unspecified.
Main entry points: kafkaConsumer.ipynb Data consumer · kafkaProducer.ipynb Data producer
Components 11
Streaming
- Data producer
kafkaProducer.ipynb - Apache Kafka
- Data consumer
kafkaConsumer.ipynb - AWS EC2
Data Lake
- AWS S3
- Glue crawler
- Glue catalog
Analytics
- AWS Athena
- SQL queries
kafka queries.txt - QuickSight
Other components
- Stock dataset
indexProcessed.csv
Connections 9
- Data producer → Apache Kafka: publishes data
README.md - Data consumer → Apache Kafka: consumes data
README.md - Data consumer → AWS S3: stores CSV
README.md - Glue crawler → AWS S3: crawls data
README.md - Glue crawler → Glue catalog: builds metadata
README.md - AWS Athena → Glue catalog: queries cataloged data
README.md - AWS Athena → AWS S3: queries stored data
README.md - AWS Athena → QuickSight: enables visualization
README.md - AWS EC2 → Apache Kafka: hosts Kafka
README.md