# jaehyeon-kim/flink-demos architecture

> Architecture diagram of the GitHub repository jaehyeon-kim/flink-demos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jaehyeon-kim/flink-demos
- Repository: https://github.com/jaehyeon-kim/flink-demos

## Overview

This repository is a collection of Apache Flink demos and learning exercises, not one integrated application. Its principal workflows show data entering through Kafka, files, or custom sources; being transformed with DataStream or Table API operations; and producing aggregates or other results in sinks. The graph keeps the README’s distinct projects and capabilities separate. Some connections are documented only in the README, so unsampled implementations are represented at their existing project-directory paths rather than inferred in detail.

Main entry points: [`real-time-streaming-aws/producer/app.py`](https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/producer/app.py) (Taxi Producer), [`pyflink-udemy/s4_11_session_windows_sql.py`](https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s4_11_session_windows_sql.py) (Session Windows), [`fraud-detection/local/processor.py`](https://github.com/jaehyeon-kim/flink-demos/blob/master/fraud-detection/local/processor.py) (Local Fraud App), [`fraud-detection/remote/processor.py`](https://github.com/jaehyeon-kim/flink-demos/blob/master/fraud-detection/remote/processor.py) (Managed Fraud App)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_aws["AWS Pipelines"]
  node_producer["Taxi Producer<br/>[app.py]"]
  node_kafka["Kafka Cluster"]
  node_aws_flows["Taxi Flink Flows"]
  node_aws_services[("AWS Data Services")]
  node_lambda_consumer["Kafka Consumer<br/>[app.py]"]
  node_fraud_local["Local Fraud App<br/>[processor.py]"]
  node_fraud_remote["Managed Fraud App<br/>[processor.py]"]
  node_fraud_data["Fraud Topics"]
  node_dynamo[("DynamoDB")]
  node_window_local["Local Window App<br/>[processor.py]"]
  node_window_remote["Managed Window App<br/>[processor.py]"]
  node_window_topics["Kafka Topics"]
  node_ride_records["Taxi Ride Records<br/>[app.py]"]
end

subgraph group_pyflink["PyFlink Learning"]
  node_building["DataStream Course"]
  node_table_tutorial["PyFlink Tutorials"]
end

subgraph group_book["Book Examples"]
  node_sensor_app["Sensor Aggregation"]
  node_sensor_source["Sensor Source<br/>[SensorSource.kt]"]
  node_book_examples["DataStream Examples"]
  node_timers_windows["Time And Windows"]
  node_state_examples["State Examples"]
  node_connectors["Custom Connectors"]
  node_py_book["PyFlink Book Port"]
  node_sensor_results["Average Readings"]
end

subgraph group_courses["Course Exercises"]
  node_udemy_sources["Table Sources"]
  node_udemy_ops["Table Operations"]
  node_session_window["Session Windows"]
  node_sql_udfs["SQL UDF<br/>[python_udf.py]"]
  node_sales_topic["Sales Items Topic"]
  node_processed_sales["Processed Sales Topic"]
end

node_external_user(("Demo User"))

node_external_user -->|"runs"| node_producer
node_producer -->|"generates"| node_ride_records
node_producer -->|"publishes"| node_kafka
node_kafka -.->|"feeds"| node_aws_flows
node_aws_flows -.->|"moves data"| node_aws_services
node_kafka -.->|"triggers"| node_lambda_consumer
node_fraud_local -.->|"reads topics"| node_fraud_data
node_fraud_remote -.->|"reads topics"| node_fraud_data
node_fraud_local -.->|"writes flagged"| node_dynamo
node_fraud_remote -.->|"writes flagged"| node_dynamo
node_window_local -.->|"reads and writes"| node_window_topics
node_window_remote -.->|"reads and writes"| node_window_topics
node_sensor_app -->|"creates source"| node_sensor_source
node_sensor_app -->|"computes"| node_sensor_results
node_external_user -->|"studies"| node_building
node_external_user -->|"studies"| node_table_tutorial
node_external_user -->|"studies"| node_py_book
node_external_user -->|"studies"| node_book_examples
node_external_user -->|"studies"| node_timers_windows
node_external_user -->|"studies"| node_state_examples
node_external_user -->|"studies"| node_connectors
node_external_user -->|"studies"| node_udemy_sources
node_external_user -->|"studies"| node_udemy_ops
node_external_user -->|"studies"| node_session_window
node_external_user -->|"studies"| node_sql_udfs
node_sales_topic -->|"supplies rows"| node_session_window
node_session_window -->|"writes aggregate"| node_processed_sales

click node_producer "https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/producer/app.py"
click node_aws_flows "https://github.com/jaehyeon-kim/flink-demos/tree/master/real-time-streaming-aws"
click node_lambda_consumer "https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/consumer/app.py"
click node_fraud_local "https://github.com/jaehyeon-kim/flink-demos/blob/master/fraud-detection/local/processor.py"
click node_fraud_remote "https://github.com/jaehyeon-kim/flink-demos/blob/master/fraud-detection/remote/processor.py"
click node_window_local "https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-getting-started-on-aws/local/processor.py"
click node_window_remote "https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-getting-started-on-aws/remote/processor.py"
click node_sensor_app "https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt"
click node_sensor_source "https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/sensor/SensorSource.kt"
click node_book_examples "https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter5"
click node_timers_windows "https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter6"
click node_state_examples "https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter7"
click node_connectors "https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter8/CustomConnectors.kt"
click node_py_book "https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-pyflink/src"
click node_building "https://github.com/jaehyeon-kim/flink-demos/tree/master/building-pyflink-apps/src"
click node_table_tutorial "https://github.com/jaehyeon-kim/flink-demos/tree/master/pyflink-doc"
click node_udemy_sources "https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s3_02_python_source.py"
click node_udemy_ops "https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s4_01_projections.py"
click node_session_window "https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s4_11_session_windows_sql.py"
click node_sql_udfs "https://github.com/jaehyeon-kim/flink-demos/blob/master/sql-cookbook/src/python_udf.py"
click node_ride_records "https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/producer/app.py"
click node_sensor_results "https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_kafka,node_aws_flows,node_aws_services,node_lambda_consumer,node_fraud_local,node_fraud_remote,node_fraud_data,node_dynamo,node_window_local,node_window_remote,node_window_topics,node_ride_records,node_external_user toneBlue
class node_building,node_table_tutorial toneAmber
class node_sensor_app,node_sensor_source,node_book_examples,node_timers_windows,node_state_examples,node_connectors,node_py_book,node_sensor_results toneMint
class node_udemy_sources,node_udemy_ops,node_session_window,node_sql_udfs,node_sales_topic,node_processed_sales toneRose
```

## Components

### AWS Pipelines

- [Taxi Producer](https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/producer/app.py): `real-time-streaming-aws/producer/app.py`
- **Kafka Cluster**
- [Taxi Flink Flows](https://github.com/jaehyeon-kim/flink-demos/tree/master/real-time-streaming-aws): `real-time-streaming-aws`
- **AWS Data Services**
- [Kafka Consumer](https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/consumer/app.py): `real-time-streaming-aws/consumer/app.py`
- [Local Fraud App](https://github.com/jaehyeon-kim/flink-demos/blob/master/fraud-detection/local/processor.py): `fraud-detection/local/processor.py`
- [Managed Fraud App](https://github.com/jaehyeon-kim/flink-demos/blob/master/fraud-detection/remote/processor.py): `fraud-detection/remote/processor.py`
- **Fraud Topics**
- **DynamoDB**
- [Local Window App](https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-getting-started-on-aws/local/processor.py): `pyflink-getting-started-on-aws/local/processor.py`
- [Managed Window App](https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-getting-started-on-aws/remote/processor.py): `pyflink-getting-started-on-aws/remote/processor.py`
- **Kafka Topics**
- [Taxi Ride Records](https://github.com/jaehyeon-kim/flink-demos/blob/master/real-time-streaming-aws/producer/app.py): `real-time-streaming-aws/producer/app.py`

### PyFlink Learning

- [DataStream Course](https://github.com/jaehyeon-kim/flink-demos/tree/master/building-pyflink-apps/src): `building-pyflink-apps/src`
- [PyFlink Tutorials](https://github.com/jaehyeon-kim/flink-demos/tree/master/pyflink-doc): `pyflink-doc`

### Book Examples

- [Sensor Aggregation](https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt`
- [Sensor Source](https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/sensor/SensorSource.kt): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/sensor/SensorSource.kt`
- [DataStream Examples](https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter5): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter5`
- [Time And Windows](https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter6): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter6`
- [State Examples](https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter7): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter7`
- [Custom Connectors](https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter8/CustomConnectors.kt): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter8/CustomConnectors.kt`
- [PyFlink Book Port](https://github.com/jaehyeon-kim/flink-demos/tree/master/stream-processing-with-pyflink/src): `stream-processing-with-pyflink/src`
- [Average Readings](https://github.com/jaehyeon-kim/flink-demos/blob/master/stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt): `stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt`

### Course Exercises

- [Table Sources](https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s3_02_python_source.py): `pyflink-udemy/s3_02_python_source.py`
- [Table Operations](https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s4_01_projections.py): `pyflink-udemy/s4_01_projections.py`
- [Session Windows](https://github.com/jaehyeon-kim/flink-demos/blob/master/pyflink-udemy/s4_11_session_windows_sql.py): `pyflink-udemy/s4_11_session_windows_sql.py`
- [SQL UDF](https://github.com/jaehyeon-kim/flink-demos/blob/master/sql-cookbook/src/python_udf.py): `sql-cookbook/src/python_udf.py`
- **Sales Items Topic**
- **Processed Sales Topic**

### Other components

- **Demo User**

## Connections

- Demo User → Taxi Producer: runs
- Taxi Producer → Taxi Ride Records: generates
- Taxi Producer → Kafka Cluster: publishes
- Kafka Cluster → Taxi Flink Flows: feeds
- Taxi Flink Flows → AWS Data Services: moves data
- Kafka Cluster → Kafka Consumer: triggers
- Local Fraud App → Fraud Topics: reads topics
- Managed Fraud App → Fraud Topics: reads topics
- Local Fraud App → DynamoDB: writes flagged
- Managed Fraud App → DynamoDB: writes flagged
- Local Window App → Kafka Topics: reads and writes
- Managed Window App → Kafka Topics: reads and writes
- Sensor Aggregation → Sensor Source: creates source
- Sensor Aggregation → Average Readings: computes
- Demo User → DataStream Course: studies
- Demo User → PyFlink Tutorials: studies
- Demo User → PyFlink Book Port: studies
- Demo User → DataStream Examples: studies
- Demo User → Time And Windows: studies
- Demo User → State Examples: studies
- Demo User → Custom Connectors: studies
- Demo User → Table Sources: studies
- Demo User → Table Operations: studies
- Demo User → Session Windows: studies
- Demo User → SQL UDF: studies
- Sales Items Topic → Session Windows: supplies rows
- Session Windows → Processed Sales Topic: writes aggregate