How rafaeljyokondostudy/spark-realtime-streaming works
This project streams review records from a JSON dataset over TCP into a Spark Structured Streaming job, which parses and enriches them with OpenAI sentiment labels before publishing to Kafka. The README also depicts Kafka Connect, Elasticsearch, and Kibana downstream; those are documented topology, not implementations in this tree. The review schema file exists, but the sampled Spark job defines its parsing schema inline and does not import that file. The dataset itself is not present in the tree.
Main entry points: spark-streaming.py Spark Stream Consumer · streaming-socket.py TCP Socket Producer · config.py API Settings · reviews.schema.avsc.py Review Record Schema
Components 10
Review Ingestion
- TCP Socket Producer
src/jobs/streaming-socket.py
Stream Processing
- Spark Stream Consumer
src/jobs/spark-streaming.py - API Settings
src/config/config.py
External Services
- OpenAI API
- Kafka Topic
- Kafka Connect Sink
- Elasticsearch
- Kibana
Review Schema
- Review Record Schema
src/schemas/reviews.schema.avsc.py
Other components
- JSON Review Dataset
Connections 9
- JSON Review Dataset → TCP Socket Producer: reads records
README.md - TCP Socket Producer → Spark Stream Consumer: streams TCP
src/jobs/streaming-socket.py - Spark Stream Consumer → Review Record Schema: parses reviews
- Spark Stream Consumer → API Settings: loads settings
src/jobs/spark-streaming.py - Spark Stream Consumer → OpenAI API: classifies text
src/jobs/spark-streaming.py - Spark Stream Consumer → Kafka Topic: publishes records
src/jobs/spark-streaming.py - Kafka Topic → Kafka Connect Sink: feeds sink
README.md - Kafka Connect Sink → Elasticsearch: indexes records
README.md - Elasticsearch → Kibana: visualizes data
README.md