# rafaeljyokondostudy/spark-realtime-streaming architecture

> Architecture diagram of the GitHub repository rafaeljyokondostudy/spark-realtime-streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rafaeljyokondostudy/spark-realtime-streaming
- Repository: https://github.com/rafaeljyokondostudy/spark-realtime-streaming

## Overview

This project streams review records from a JSON dataset over TCP into a Spark Structured Streaming job, which parses and enriches them with OpenAI sentiment labels before publishing to Kafka. The README also depicts Kafka Connect, Elasticsearch, and Kibana downstream; those are documented topology, not implementations in this tree. The review schema file exists, but the sampled Spark job defines its parsing schema inline and does not import that file. The dataset itself is not present in the tree.

Main entry points: [`src/jobs/spark-streaming.py`](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/jobs/spark-streaming.py) (Spark Stream Consumer), [`src/jobs/streaming-socket.py`](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/jobs/streaming-socket.py) (TCP Socket Producer), [`src/config/config.py`](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/config/config.py) (API Settings), [`src/schemas/reviews.schema.avsc.py`](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/schemas/reviews.schema.avsc.py) (Review Record Schema)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Review Ingestion"]
  node_producer["TCP Socket Producer"]
end

subgraph group_processing["Stream Processing"]
  node_spark["Spark Stream Consumer<br/>[spark-streaming.py]"]
  node_settings["API Settings<br/>[config.py]"]
end

subgraph group_integration["External Services"]
  node_openai["OpenAI API"]
  node_kafka["Kafka Topic"]
  node_connect["Kafka Connect Sink"]
  node_elasticsearch[("Elasticsearch")]
  node_kibana["Kibana"]
end

subgraph group_schema["Review Schema"]
  node_review_schema["Review Record Schema"]
end

node_dataset["JSON Review Dataset"]

node_dataset -->|"reads records"| node_producer
node_producer -->|"streams TCP"| node_spark
node_spark -.->|"parses reviews"| node_review_schema
node_spark -->|"loads settings"| node_settings
node_spark -->|"classifies text"| node_openai
node_spark -->|"publishes records"| node_kafka
node_kafka -->|"feeds sink"| node_connect
node_connect -->|"indexes records"| node_elasticsearch
node_elasticsearch -->|"visualizes data"| node_kibana

click node_producer "https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/jobs/streaming-socket.py"
click node_spark "https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/jobs/spark-streaming.py"
click node_settings "https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/config/config.py"
click node_review_schema "https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/schemas/reviews.schema.avsc.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_dataset toneBlue
class node_spark,node_settings toneAmber
class node_openai,node_kafka,node_connect,node_elasticsearch,node_kibana toneMint
class node_review_schema toneRose
```

## Components

### Review Ingestion

- [TCP Socket Producer](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/jobs/streaming-socket.py): `src/jobs/streaming-socket.py`

### Stream Processing

- [Spark Stream Consumer](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/jobs/spark-streaming.py): `src/jobs/spark-streaming.py`
- [API Settings](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/config/config.py): `src/config/config.py`

### External Services

- **OpenAI API**
- **Kafka Topic**
- **Kafka Connect Sink**
- **Elasticsearch**
- **Kibana**

### Review Schema

- [Review Record Schema](https://github.com/rafaeljyokondostudy/spark-realtime-streaming/blob/main/src/schemas/reviews.schema.avsc.py): `src/schemas/reviews.schema.avsc.py`

### Other components

- **JSON Review Dataset**

## Connections

- JSON Review Dataset → TCP Socket Producer: reads records (evidence: `README.md`)
- TCP Socket Producer → Spark Stream Consumer: streams TCP (evidence: `src/jobs/streaming-socket.py`)
- Spark Stream Consumer → Review Record Schema: parses reviews
- Spark Stream Consumer → API Settings: loads settings (evidence: `src/jobs/spark-streaming.py`)
- Spark Stream Consumer → OpenAI API: classifies text (evidence: `src/jobs/spark-streaming.py`)
- Spark Stream Consumer → Kafka Topic: publishes records (evidence: `src/jobs/spark-streaming.py`)
- Kafka Topic → Kafka Connect Sink: feeds sink (evidence: `README.md`)
- Kafka Connect Sink → Elasticsearch: indexes records (evidence: `README.md`)
- Elasticsearch → Kibana: visualizes data (evidence: `README.md`)