# dhia4884/tp06-real-time-processing architecture

> Architecture diagram of the GitHub repository dhia4884/tp06-real-time-processing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhia4884/tp06-real-time-processing
- Repository: https://github.com/dhia4884/tp06-real-time-processing

## Overview

This repository demonstrates two independent real-time processing workflows: campus events are produced to Kafka and aggregated by Spark, while department check-ins are produced to Kafka and aggregated with Flink SQL. The graph preserves the documented parsing, event-time windowing, watermark, and aggregation stages. Flink SQL files are represented from the README and tree; their contents were not sampled, so those relationships rely on README evidence. Kafka is an external service, and the graph does not infer an unsampled Flink job launcher.

Main entry points: [`Lab-B-Flink-SQL/sql/create_table.sql`](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/sql/create_table.sql) (Check-ins Table), [`Lab-A-Spark-Streaming/spark_streaming.py`](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py) (Kafka Ingestion), [`Lab-B-Flink-SQL/sql/department_checkins.sql`](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/sql/department_checkins.sql) (Department Windows), [`Lab-A-Spark-Streaming/producer.py`](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/producer.py) (Campus Producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_lab_a["Spark Streaming"]
  node_campus_producer["Campus Producer<br/>[producer.py]"]
  node_spark_ingest["Kafka Ingestion<br/>[spark_streaming.py]"]
  node_spark_parse["Parse Events<br/>[spark_streaming.py]"]
  node_spark_aggregate["Window Counts<br/>[spark_streaming.py]"]
  node_spark_output["Console Results<br/>[spark_streaming.py]"]
end

subgraph group_lab_b["Flink SQL"]
  node_checkin_producer["Check-in Producer<br/>[producer_flink.py]"]
  node_flink_table["Check-ins Table<br/>[create_table.sql]"]
  node_flink_query["Department Windows"]
  node_flink_connector["Kafka Connector"]
  node_flink_results["Check-in Results"]
end

subgraph group_messaging["Kafka Messaging"]
  node_kafka["Kafka Broker"]
end

node_campus_producer -->|"sends events"| node_kafka
node_kafka -->|"streams topic"| node_spark_ingest
node_spark_ingest -->|"parses JSON"| node_spark_parse
node_spark_parse -->|"groups events"| node_spark_aggregate
node_spark_aggregate -->|"writes updates"| node_spark_output
node_checkin_producer -->|"sends check-ins"| node_kafka
node_kafka -->|"reads topic"| node_flink_table
node_flink_table -->|"queries table"| node_flink_query
node_flink_query -->|"produces counts"| node_flink_results
node_flink_connector -.->|"enables Kafka"| node_flink_table

click node_campus_producer "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/producer.py"
click node_spark_ingest "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py"
click node_spark_parse "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py"
click node_spark_aggregate "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py"
click node_spark_output "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py"
click node_checkin_producer "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/producer_flink.py"
click node_flink_table "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/sql/create_table.sql"
click node_flink_query "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/sql/department_checkins.sql"
click node_flink_connector "https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/lib/flink-sql-connector-kafka-3.3.0-1.20.jar"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_campus_producer,node_spark_ingest,node_spark_parse,node_spark_aggregate,node_spark_output toneBlue
class node_checkin_producer,node_flink_table,node_flink_query,node_flink_connector,node_flink_results toneAmber
class node_kafka toneMint
```

## Components

### Spark Streaming

- [Campus Producer](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/producer.py): `Lab-A-Spark-Streaming/producer.py`
- [Kafka Ingestion](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py): `Lab-A-Spark-Streaming/spark_streaming.py`
- [Parse Events](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py): `Lab-A-Spark-Streaming/spark_streaming.py`
- [Window Counts](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py): `Lab-A-Spark-Streaming/spark_streaming.py`
- [Console Results](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-A-Spark-Streaming/spark_streaming.py): `Lab-A-Spark-Streaming/spark_streaming.py`

### Flink SQL

- [Check-in Producer](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/producer_flink.py): `Lab-B-Flink-SQL/producer_flink.py`
- [Check-ins Table](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/sql/create_table.sql): `Lab-B-Flink-SQL/sql/create_table.sql`
- [Department Windows](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/sql/department_checkins.sql): `Lab-B-Flink-SQL/sql/department_checkins.sql`
- [Kafka Connector](https://github.com/dhia4884/tp06-real-time-processing/blob/main/Lab-B-Flink-SQL/lib/flink-sql-connector-kafka-3.3.0-1.20.jar): `Lab-B-Flink-SQL/lib/flink-sql-connector-kafka-3.3.0-1.20.jar`
- **Check-in Results**

### Kafka Messaging

- **Kafka Broker**

## Connections

- Campus Producer → Kafka Broker: sends events (evidence: `Lab-A-Spark-Streaming/producer.py`)
- Kafka Broker → Kafka Ingestion: streams topic (evidence: `Lab-A-Spark-Streaming/spark_streaming.py`)
- Kafka Ingestion → Parse Events: parses JSON (evidence: `Lab-A-Spark-Streaming/spark_streaming.py`)
- Parse Events → Window Counts: groups events (evidence: `Lab-A-Spark-Streaming/spark_streaming.py`)
- Window Counts → Console Results: writes updates (evidence: `Lab-A-Spark-Streaming/spark_streaming.py`)
- Check-in Producer → Kafka Broker: sends check-ins (evidence: `Lab-B-Flink-SQL/producer_flink.py`)
- Kafka Broker → Check-ins Table: reads topic (evidence: `README.md`)
- Check-ins Table → Department Windows: queries table (evidence: `README.md`)
- Department Windows → Check-in Results: produces counts (evidence: `README.md`)
- Kafka Connector → Check-ins Table: enables Kafka
